x264farm: distributed encoding [Archive] - Page 3

St Devious

20th December 2007, 00:54

ok i'm giving this one more try . I selected x264 farm codec in megui and i see these options which r different from x264 .

I selected HQ-slowest profile , and left all other parameters at default . So wat do i do now , copy the collective command line parameters which r written in box near bottom or can i encode to x264farm using megui .

If yes , how do i set up the agents ? run megui too on them ?

Another question : I want to have only one agent . So in the config.xml file of the controller , do i have 2 agents ( 1 my pc and 2nd another pc ) or only the agent's ip address and other info .

Some more questions : Where do i put nice.exe and avs2yuv.exe ?

I tried running the controller and agent . agent is running fine , but controller exits . here r the config.xml files for both .

CONTROLLER :

<config>
<temp>C:\x264farm\controller\temp</temp> 

<agents>

<agent name="Matrix"> 
<ip>192.168.111.147</ip> 
<port from="40700" to="40703"/> 
<number>2</number> 
</agent>

<agent name="Agent Smith">
<ip>192.168.111.66</ip>
<port from="40700" to="40703"/>
<number pad="1">1</number>

</agent>

</agents>
<adhoc controller="12345" agent="23456"/>
</config>

AGENT :

<config>

<temp>C:\x264farm\agent\temp</temp>


<port from="40700" to="40703"/>



<name>Local</name>
<number pad="0">2</number>


<x264>x264</x264>


<nice>20</nice>


<base></base>


<agentpipe>0</agentpipe>


<compression>
<type>0</type>
<type>1</type>
</compression>




<adhoc controller="12345" agent="23456"/>
</config>

And the error i get is :

http://i16.tinypic.com/72ktsvs.jpg

audyovydeo

20th December 2007, 09:42

Again, I haven't tried the megui option. But what is your controller command-line ?

to resume :

0. read through the doc and this thread once again.

1. deploy the whole cast of characters (directories, paths, all executables, source avi, avs scripts, etc). Nice and avi2yuv should, as always with any windows program, somewhere in your path. just like the correct version of x264.exe

2. edit the xml files accordingly (temp paths are important and finicky to configure)

3. launch the agent(s)

4. run the controller
here is my cmdline :

controller.exe -B 500kbps --avs temp\input.avs -o test.mkv -1 "--crf 25 -r 1 -b 3 -A none -m 1 --me hex" -2 "-r 1 -b 3 -A none -m 1 --me hex"

and ... (how do they say ?) ... et voila' !

cheers
audyovydeo

SpAwN_gUy

20th December 2007, 09:50

I selected HQ-slowest profile , and left all other parameters at default . So wat do i do now , copy the collective command line parameters which r written in box near bottom or can i encode to x264farm using megui . and why do you think (and asked about) this Mod is made for?
and why x264farm is "Codec" in meGUI ? ...

If yes , how do i set up the agents ? run megui too on them ? NO. you don't need to setUp meGUI on them..
just run agent.exe on agentPC and let it run..

Another question : I want to have only one agent . So in the config.xml file of the controller , do i have 2 agents ( 1 my pc and 2nd another pc ) or only the agent's ip address and other info . well.. since we have adhoc feature (and it IS working for me at least).. you dont even need to specify agents in controller's config.. but you can leave it like that...

so.. the only mashines that will be Encoding are the PCs where agent.exe is started.. and RUNNING

Some more questions : Where do i put nice.exe and avs2yuv.exe ? agent's and controller's folder

I tried running the controller and agent . agent is running fine , but controller exits . here r the config.xml files for both. nice picture you have here.. :) ..
this picture means that you haven't read anything.. i'm sorry.. but "WHY???"
READ the manual.. on how to use x264farm...
controller-based encoding is the simpliest thing to set up.. :readfaq:

SpAwN_gUy

20th December 2007, 10:37

But what is your controller command-line ?according to the picture he showed.. it's "nothing" :) ..

BWAHAHAHA.... the previous post was my 200th ;) ... nice using :)

St Devious

20th December 2007, 14:50

@ Spawn_guy : u r not exactly helping me here .

I gave u my agent and controller config , tell me wats wrong with that if u really want to help rather than make smart remarks . I need help , thats y i posted here.

So i select megui mod on the controller pc , and run agent.exe on each of the agent pc's and start encoding in megui ?

Look at the picture and my agent , controller config and tell y the controller cant find agent pc .

The ip's of boh the pc's r the ip's on lan , is that correct ?

Sharktooth

20th December 2007, 15:10

im sorry to say this, but your problems (as well as in the other threads) are all related to your ignorance.
those kind of things (video encoding) are not simple and you need to read and understand how things work.
do as spawn guy said: READ.
If you find it too much complicated, then start with the basic things.

audyovydeo

20th December 2007, 15:12

Hard to help someone who's not putting effort into the task. We've practically spoon-fed you all the info.

Specifically :

1. in your agent XML file, the fact that you got < nice> value of 20 only shows that you have not read this post :

http://forum.doom9.org/showthread.php?p=1068890#post1068890

2. in your controller xml there is no trace of the ADHOC directive, which means you havent read this post :

http://forum.doom9.org/showthread.php?p=1068730#post1068730

3. your controller command-line is non-existent, which means you haven't read the documentation.

Hard to figure what else we can do, short of drilling for oil.

cheers
audyovydeo

SpAwN_gUy

20th December 2007, 16:26

2. in your controller xml there is no trace of the ADHOC directive, which means you havent read this post : well.. i guess.. he has "adhoc" enabled.. in both configs..

'kay.. my give up..
-1) setUP configs.. i guess everything seems to be fine
0) run agent.exe on agentPC's
1) configure in meGUI path to the controller in options..
-----------------READTHEGUIDEONmeGUIUSAGE--------
2) create an AVS and Choose it as an input
3) choose x264farm as Codec
4) configure params as in the post i gave you
5) press OK.
6) push Enque
7) press start on the second Tab in meGUI..
wait

i guess i'll need some money for such kind of things :) ..
do we have to think for you?

i guess there will be no posts from me in 4 days.. i'm going to Geneva... for 2 weeks.. anyone want's to meet me there? :)

UPD: ah.. BTW.. there is one more tool for x264farm ... i wonder what it's name.. i dunno is it upDated or not.. you can search for it in here.. i tried it a long time ago.. so search fo x264farm support

Sharktooth

20th December 2007, 16:34

Spawn_guy, Geneva? No... for now, my best would be Genova (Italy).

SpAwN_gUy

21st December 2007, 08:27

Spawn_guy, Geneva? No... for now, my best would be Genova (Italy).
hm.. it's quite far from Geneva (Switzerland) .. :) .. i think.. but who knows :) ...

antiOFFTOP: my next steps on x264farm.. are:
1) organize storing normal x264settings in meGUI for x264farm (now they are only "strings")
2) complete my agentGUI (Delphi).. (i'm waiting for a new updated version of x264farm.. as omion said.. he's trying to release it soon)
2b) write an "aget as service starter".. not all of my people are satisfied with using it as a gui... (they are working in Terminals)
3) write and agentGUI/Service installer.. so everyone could change sources and compile the installer.. and then to share it with others.. who wishes to participate in encoding..

agentGUI is allmost complete.. i'm reWriting it now.. and most of my people use an old one..

but.. as tomorrow is my last workDay this year.. i suppose.. neXt year..

ChrisBensch

3rd January 2008, 05:29

SpAwN_gUy

6th January 2008, 20:17

Is it possible, since we don't really have single-pass yet, to have a REALLY fast first pass, then use our second pass as the "real" pass? Maybe just dumping the first pass data to /dev/null? I use the crf preset of MeGUI in my cli so I'm not sure if a first pass even benefits me at all. Right now it's just running the same single pass, just twice. I wish there were a better way.
Yep.. i'm allmost back from... where i was :) ...

omion is currently working on "single-pass" encoding... (as i've heard.. well.. that was last year... so, maybe, soon)
and "yes" for now.. using CRF(for x264) and PERCENTAGE(for x264farm) will do Two passes... but during the first one it will only write .stats..

ChrisBensch

11th January 2008, 02:56

omion

11th January 2008, 09:14

Hi guys! I'm back!

Is it possible, since we don't really have single-pass yet, to have a REALLY fast first pass, then use our second pass as the "real" pass? Maybe just dumping the first pass data to /dev/null? I use the crf preset of MeGUI in my cli so I'm not sure if a first pass even benefits me at all. Right now it's just running the same single pass, just twice. I wish there were a better way.

I don't really know what you are saying here. The first pass video data is actually sent to /dev/null (check the code!); it only keeps the stats file. The stats file is basically what sets the second pass in the right direction. I could make a one-pass option (and am still thinking about it, as spawn_guy says) but it would require an entire rewrite of the encoding loop.

This might be a dumb question, but is there a sort of "rule of thumb" for percentages and how they correlate to a CRF value? I usually encode with a CRF of 18...is there a percentage that would match that?
Yup. 100% gets you the same file size that the first pass would have gotten, had you just done a one-pass encode. So if you use --crf 18 on the first pass, 100% will get you around crf 18 on the second pass (but hopefully with higher quality, which is the whole point of a two-pass encode).

Now, on an unrelated note:
<rant>
Networking is HORRIBLE.
I've spent the larger part of two months working on the stupid networking code. I've made about 4 different implementations of it, and each one had some sort of horrible problem with it that would make it completely unacceptable. My current code actually crashes the program. Fairly reliably, too. OCaml code is not supposed to crash. Exit with an error, maybe, but not crash. I don't know how I managed it. It was probably due to some workaround that I added a month ago because Windows can't seem to read and write to the same socket at the same time. So now I have to think of another workaround. The main problem is probably just Windows, which really bugs me because I know those problems aren't going to be fixed. Should I tell the Windows devs to implement timeouts on send/receive functions? The best that can happen is they add it to "Windows 7" when it comes out in 5 years.

I've been working on this code for quite a while now. I think I can do it with two uni-directional sockets between the agent and the controller. But they can't be totally uni-directional, or else the TCP "delayed ack" algorithm will kick in and bring everything down to 10kbps. WHY is that stupid "congestion control" enabled on so many computers? Is an extra 60 bytes really going to take down the network?

I think the code is getting to be too much for me to handle. In order to do all of these workarounds I have to make a whole bunch of threads to shuffle around the data, and the networking makes it break sometimes. I'll get errors like "Thread 37 failed with ENOTSOCK" Not a socket? A half second ago it was a socket, and it was working just fine. And what on earth is thread 37? Is that the thread that is sending data? Or the thread that's receiving data? Or maybe it's the thread that sends dummy data on the loopback socket that I had to set up in order to get Windows unstuck from the select statement that works just fine on every other operating system. So I track it down, and it turns out it's on the thread that sends a ping signal to the controller as long as the 3 threads that communicate with x264 are still working... so I have to workaround that too.

All this extra junk to try and get computers to work together. It's really aggrivating.
</rant>

So, how was your Christmas? :rolleyes:

SpAwN_gUy

11th January 2008, 09:22

Alright, so percentage is the way for farm. This might be a dumb question, but is there a sort of "rule of thumb" for percentages and how they correlate to a CRF value? I usually encode with a CRF of 18...is there a percentage that would match that?
the History Chanel:
at first.. first-pass settings for x264farm needed NO "bitrate" or "crf" options.. 'cause in older x264 builds the default option, when no parameter was specified was "--crf 26" .. or 25..
but after.. the default setting was removed.. and thats why i had to add it manually to comLine Generation in meGUI-Mod... but i never tried such low bitrates so.. my value is "--crf 18" for the first pass..
so.. basically, when you specify in first-pass settings "--crf 18" and for the x264farm bitrate "100" (or "100%") then you will have an TwoPass Encoding with bitrate of CRF 18 ...

on how "percentage" of bitrate is assumed you can read somewhere here or i guess it was somewhere in the doc's, provided with binaries

------------
UPD: glad you're back omion :) ...
So, how was your Christmas? :rolleyes: very nice indeed.. Geneva seems fine.. Berlin is nice at night.. and Prague is extremely Cheap.. and Beer is beautiful :)

Elvis'es and mine Birthsday went just fine :) .. i got drunk.. again.. aswell as on celebration of NY in my Corporation :)

and Testings of DGAVCDec (neuron2's) with x264farm.. just ended with beautiful results :) ... (yeah, i had to modify internals of ".dga" file aswell :) ...)

30th January 2008, 16:02

Hello guys!
I'm trying to encode tga-sequence (1500frames long), so it has no GOPs by default :)
Which settings should I use to speed-up fist-pass calculations (I mean 1st pass split parameters)?

Poopoo

30th January 2008, 16:56

very nice indeed.. Geneva seems fine.. Berlin is nice at night.. and Prague is extremely Cheap.. and Beer is beautiful :)

You drank beer in Prague ? How long have you been sick from it ?

omion

1st February 2008, 23:21

Hello guys!
I'm trying to encode tga-sequence (1500frames long), so it has no GOPs by default :)
Which settings should I use to speed-up fist-pass calculations (I mean 1st pass split parameters)?

If it's all smooth motion (no clear GOPs) and fairly long then you can change the --fastavs parameter to something like a small black clip (must be the same # of frames and framerate).

HOWEVER (and this is a big however) if there are no clear GOP boundaries in the entire clip, x264farm will NOT split up the file at all. This means that the first pass will encode a bit slower than if you didn't use x264farm. The problem is that x264 will encode a bunch of GOPs of length 250 (by default) and so any encode that does not start on a multiple of 250 will be thrown out. x264farm decides to "save" computing resources until it can find a reasonable split point.

So in your case there is not much you can do to speed up the first pass. Actually, the default job length is longer than 1500 frames, so x264farm won't even try to split it.

@all:
I suppose this is as good a time as any to give a little update. I've decided to concentrate my time on making a single-pass x264farm, rather than trying to hack my way through the mess that is the current 2-pass code. It will be highly simplified (no wacky split options, config.xml no longer required, smaller executables, etc.) but it should be just as powerful as - and hopefully more efficient than - the current program. (And I'm making a 64-bit version, not that it would make any speed difference at all:p)

I'll probably end up applying the algorithms to a new 2-pass x264farm, but that will be a while away.

ChrisBensch

11th February 2008, 06:16

Any news on the MeGui-mod for x264farm? How about a client config gui? I dabble in Java/C# and I'd be happy to contribute if I can.

SpAwN_gUy

11th February 2008, 11:16

Any news on the MeGui-mod for x264farm? How about a client config gui? I dabble in Java/C# and I'd be happy to contribute if I can.
i'm busy with my work.. i'm hacking linux.. ;) ..
so, no updates for now :( ... contact me via PM... for more instructions :) ...

konzerten_nr9

10th March 2008, 04:55

I am runnig an x264farm consists of a controller and 6 agents, 5 agents running on Windows using agent-based encoding, 1 on Linux using controller-based encoding.

Now I am runnig a test with a simple test.avs as input ike:
directshowsource("test.wmv", audio=false)
which test.wmv has 9440 frames.

My command line is:
controller -B 3500kbps --qcomp 0.75 -1 "--crf 19 --keyint 25 --min-keyint 25 --bframes 1 --bime --level 41 --direct auto --mixed-refs --b-pyramid --weightb --8x8dct --no-psnr --no-ssim --analyse all --threads auto --progress" -2 "--keyint 25 --min-keyint 25 --bframes 1 --bime --level 41 --direct auto --mixed-refs --b-pyramid --weightb --8x8dct --no-psnr --no-ssim --subme 5 --ref 1 --trellis 1 --me hex --merange 16 --threads auto --progress" --avs ..\test.avs -o ..\test.mkv

The problem is, the footage is splitted into several parts during the first pass, but it is not being encoded by multiple agents simultaneously. Instead, the splitted parts are encoded by a single agent sequentially.

This results in the throughput of first pass varying between 2 to 15fps depending on the agent picked up at the beginning of the first pass, while the throughput of the second pass is around 30fps.

Is there any way to have the first pass encoded by multiple agents simultaneously?

Thank you in advance for your kind suggestions.

Jabroni

10th March 2008, 05:00

konzerten_nr9

10th March 2008, 06:12

Hello Jabroni,

If you scroll on this thread I asked the same thing.. the problem is that its not able to do more splitting because it cant find more I-Frames (or whatever type of frame it is used to split).

Actually the first pass is splitted into 2 parts:
Frames 0-5159 Frames 5160-9439
but all the jobs are done on 1 single agent serially.
Thus I thought it is a new situation here...

driftr

13th March 2008, 15:15

So here's my latest problem. Large source files. Like over 10 gig high bitrate source files. I often give up on x264farm because the first pass takes forever. The process to split the file into batches and send the jobs out maxes out at 100% proc usage (or 25% on my quad core system) and goes extremely slow and might get a second job out before the first one finishes encoding. Overall the first pass is so slow just doing command line x264 encoding on the single computer alone is faster. Any thoughts on how to get the process multithreaded or better handled?

SpAwN_gUy

13th March 2008, 16:04

i'm here again...

'kay.. mine things are doing fine.. even with 10Gb 1080p x264 encoded files.. did that a while ago... first pass - multi threaded..

the reason of 100% CPU on the Controller might be on Split-points determination... (i have single core.. when starting - yeah it's a bit annoying, but when working - fine..)

driftr, post you AVS script..

driftr

13th March 2008, 17:48

I wrote that rather slopily... the avs script is a simple directshowsource opening the file with no filtering or framerate change or anything. The problem is source material having sizes like 720x576 or say blu-ray with 1920x1080.

What's happening is avs2yuv can't keep up with the how fast the computers are encoding the jobs. So I suppose I could bump up the batch size but then the initial set of jobs would take really long esp. with say 15+ agents. What I'd really like to see is some kind of multiple controller setup where multiple instances of avs2yuv can create the jobs.

Right now on this simple hour long 720x576 wmv I'm just converting to x264 for fun and there's a point where agents are just idle waiting for jobs because the controller can only push out so many at a time.

SpAwN_gUy

14th March 2008, 10:38

well.. i just don't know what to say...

well.. you can run multiple controllers :) ...

and... try out setting "--batchmult 0" and/or "--rethresh 1.0"
i can't remember which one is "the one".

driftr

14th March 2008, 14:21

Humm I forgot about those settings. I can see them helping marginally but I'll play with them.

I'm not reading anything about how to run multiple controllers... got an example?

SpAwN_gUy

14th March 2008, 15:19

I'm not reading anything about how to run multiple controllers... got an example?
try to run.. 2 (two) cintroller.exe's at the same time ;) ...
like.. prepare two .bat files.. and the run them..

or make 2 Workers in meGUI-x264farm-mod ... and the run them :) ... (the old one(working) could be still found somwhere here.. my version is still in unworking state :( )

omion

17th March 2008, 22:44

Hi guys! Just to let you know, I'm not dead yet... :p

I've finally got my 1-pass x264farm in a usable state. I'll be releasing it in a few days unless some other major bugs appear.

It should be a bit more efficient than the current code, and driftr's problem should not exist (the splitter does not examine the avs file)

@driftr:
I assume that you are seeing one avs2yuv instance using 100% of one core, correct? Or is it controller.exe? If it's avs2yuv then setting "--thresh 1.0" will minimize the usage of the splitter thread at the expense of some extra time taken re-rendering frames around the split point.

Note: --rethresh is a different setting from --thresh. It's a bit complicated, but "--rethresh 1.0" will not do what you want (in fact, it will basically do all of the same calculations, then ignore them :scared: )

@SpAwN_gUy:
Do multiple controllers work? I didn't think they'd play nice with the shared stats file. I'm on my laptop right now (and lunch break is almost over) so I can't check, but did you find a way to get this to work with the current iteration of x264farm?

SpAwN_gUy

18th March 2008, 08:24

I've finally got my 1-pass x264farm in a usable state. I'll be releasing it in a few days unless some other major bugs appear.
so'.. i guess i need to stick to meGUI thread.. 'cause berrinam is now missin' ... :) :( ...

Do multiple controllers work? I didn't think they'd play nice with the shared stats file. I'm on my laptop right now (and lunch break is almost over) so I can't check, but did you find a way to get this to work with the current iteration of x264farm?Never tried that.. but i can :) .. and i will...

i was trying to say, that multiple controllers will work on different jobs (but never tried that either..)

so.. i guess.. more testing will be done :)

omion

19th March 2008, 20:10

Single-pass version is out. (Yay! A new release!)
It has a new thread (http://forum.doom9.org/showthread.php?p=1114551) to keep the bugs separate from the 2-pass version. Have fun!

vwpassion

31st March 2008, 09:24

First of all: Nice job omion!

Now my concern: Is there a way to make x264farm work with current builds of x264 from Sharktooth or older versions from Cef above 655?

Roswellian

1st April 2008, 08:26

Hello, I am newbie here and just went through the whole threads. There are too many stuff and almost totally confused me. Right now I set up the agent-base mode and my problem is that there is no response on my agents after controller started. Here is some codes on my agent side.

<config>
<temp>.\temp</temp>
<port from="40700" to="40703"/>
<name>Local</name>
<number pad="1">1</number>
<x264>x264.exe</x264>
<nice>0</nice>
<base>d:\Sample</base>
<compression>
<type>0</type>
<type>1</type>
</compression>
</config>

Controller side
<config>
<temp>.\temp</temp> 

<agents>
<agent name="Local"> 
<ip>127.0.0.1</ip> 
<port from="40700" to="40703"/> 
<number>1</number> 
</agent>

</agents>

</config>

As you see there, I tried to encode files locally by connection (One controller + One agent). My sample file is small m2ts file (5000frames) and here is my avs file.

DirectShowSource("1.GRF",video=true,fps=23.9759856527702,audio=false)
#deinterlace
crop( 0, 22, 0, -22)
Spline16Resize(1280,688) # Spline16 (Neutral)
#denoise

My controller command is like this..
controller.exe --first "--level 4.1 --threads 1 --thread-input --bframes 3 --me dia --ref 1 --subme 1 --no-dct-decimate --partitions none --progress --no-psnr --no-ssim" --second "--level 4.1 --threads 1 --bframes 3 --b-pyramid --bime --weightb --b-rdo --me umh --ref 5 --mixed-refs --subme 7 --trellis 1 --8x8dct --no-fast-pskip --progress --no-psnr --no-ssim" --avs "D:\Sample\default.avs" --output "D:\Sample\farm.mkv" --bitrate 1375kbps --batch 1000 --split 250 --thresh 20.0

after the controller is started..and log shows
.................
2008-04-01~02:15:52.78 0 @ INFINITY
2008-04-01~02:15:52.78 Queued
2008-04-01~02:15:52.78 1009 @ 13.58
2008-04-01~02:15:52.78 Empty
2008-04-01~02:15:52.78 2720 @ 48.69
2008-04-01~02:15:52.78 Queued
2008-04-01~02:15:52.78 3716 @ 17.03
2008-04-01~02:15:52.78 Empty
2008-04-01~02:15:52.78 4539 @ 13.07
2008-04-01~02:15:52.78 Queued
2008-04-01~02:15:52.78 5599 @ INFINITY
2008-04-01~02:15:52.78 0% done (0 / 5599) at 0.00 FPS
2008-04-01~02:15:52.78 Last updated: 0:18
2008-04-01~02:15:52.78 ---------------......................---------------...........---------------
2008-04-01~02:15:52.78 Agent | FPS | Description
2008-04-01~02:15:52.78 Recent errors:
2008-04-01~02:15:52.78 ~
2008-04-01~02:15:52.78 ~
2008-04-01~02:15:52.78 ~
2008-04-01~02:15:52.78 ~
2008-04-01~02:15:52.78 SPLITTER MUTEX waiting on range_list_full_condition; unlocking range_list_mutex

The controller sort of hangs there and there is also no any response on the agent window. I checked the agent port bonding by telnet, and it is fine. Since I am a newbie, it is pretty hard for me to nail down this problem. Could someone give me a little help? Will be really appreciated. :):thanks:

omion

1st April 2008, 08:41

Hello, I am newbie here and just went through the whole threads. There are too many stuff and almost totally confused me.
Me too.:eek: I'm trying to get rid of the complicated stuff in my single-pass version, but it may take a while to get into the 2-pass one...

after the controller is started..and log shows
.................
2008-04-01~02:15:52.78 0% done (0 / 5599) at 0.00 FPS
2008-04-01~02:15:52.78 Last updated: 0:18
2008-04-01~02:15:52.78 ---------------......................---------------...........---------------
2008-04-01~02:15:52.78 Agent | FPS | Description
2008-04-01~02:15:52.78 Recent errors:
2008-04-01~02:15:52.78 ~
2008-04-01~02:15:52.78 ~
2008-04-01~02:15:52.78 ~
2008-04-01~02:15:52.78 ~

It looks like the controller never gets the agent's info from the config file. It should say something about the agent right after the "Agent | FPS | Description" line. Most likely it's not using the config file you made. What is the location/name of the controller's config file?

Try adding "--config config-file-name.xml" to the controller's command line (replacing config-file-name.xml with the actual name, of course ;)) If that doesn't work, then I'll have to think of something else...

Roswellian

1st April 2008, 08:59

Omion, thanks for the quick reply. Actually I did not change config file name and config.xml is located in the same folder with agent/controller, but i still make a quick try as you mentioned. It is still not working. :-(

Edit: Just went through the whole threads again, and it was adhoc parameter problem. Now it is solved. :P

kenji123

9th April 2008, 22:15

I've integrated x264farm into my gui, i'm hopeing if someone could try it out so that i could get some feedback. its setup as controller based encoding. i've done tests using 3 pcs and it works ok. but would like to know how it works on a broader range of pcs.

file 1 - this runs on the controller pc (http://kenji123.uni.cc/ani-x264/Ani-x2641500.7z)
file 2 - copy this to all the agent pcs (http://kenji123.uni.cc/ani-x264/Ani-x264FARMClient2000.7z)

if needed, view this small tutorial to get it started; here (http://kenji123.uni.cc/ani-x264/guides/Encoding with x264Farm/index.html)

RickA

9th April 2008, 23:08

Greets,

Thank you so much for the GUI. Will give it a try as soon as my current encode finishes. I love the idea of being able to use multiple computers to work together on the same encode. :-)

Cheers,
Rick

Additional:
Forgive me for not reading all 28 pages of this thread yet. Did a quick search for x264farm and AC3 and got no hits. Since there is the option to use the MKV container can you please give us the ability to use AC3 instead of AAC for our audio? For those of us with home sound systems that prefer to decode DD5.1+ in AC3 through digital coax via soundcard passthrough. Or must we mux that in through other means? Will go back and read more before I start asking router questions. :-)

SpAwN_gUy

10th April 2008, 15:55

Forgive me for not reading all 28 pages of this thread yet. Did a quick search for x264farm and AC3 and got no hits. Since there is the option to use the MKV container can you please give us the ability to use AC3 instead of AAC for our audio? For those of us with home sound systems that prefer to decode DD5.1+ in AC3 through digital coax via soundcard passthrough. Or must we mux that in through other means?em.. how does AC3-Audio cooperates with VideoEncoding?

SpAwN_gUy

10th April 2008, 15:57

I've integrated x264farm into my gui, i'm hopeing if someone could try it out so that i could get some feedback. its setup as controller based encoding. i've done tests using 3 pcs and it works ok. but would like to know how it works on a broader range of pcs.

file 1 - this runs on the controller pc (http://kenji321.uni.cc/Ani-x2641500.7z)
file 2 - copy this to all the agent pcs (http://kenji321.uni.cc/Ani-x264FARMClient2000.7z)

if needed, view this small tutorial to get it started; here (http://kenji123.uni.cc/ani-x264/guides/Encoding with x264Farm/index.html)i've seen your GUI .. like in the start of the thread :) ... and just "before" meGUI-mod developement start..
i guess i'll take another shot.. and try it .. again :)

[P]ako

11th April 2008, 02:52

Greets,
Additional:
Forgive me for not reading all 28 pages of this thread yet. Did a quick search for x264farm and AC3 and got no hits. Since there is the option to use the MKV container can you please give us the ability to use AC3 instead of AAC for our audio? For those of us with home sound systems that prefer to decode DD5.1+ in AC3 through digital coax via soundcard passthrough. Or must we mux that in through other means? Will go back and read more before I start asking router questions. :-)

Well, x264 is just for video encoding. You still need to encode the audio with the program(s) of your liking to any of the formats that MKV supports and finally mux it with the video.

Razorholt

11th April 2008, 06:22

Date: #Apr.11.2008#
Time: [12:14 a.m.]

Stack Trace: at System.Windows.Forms.Control.MarshaledInvoke(Control caller, Delegate method, Object[] args, Boolean synchronous)
at System.Windows.Forms.Control.Invoke(Delegate method, Object[] args)
at ani_x264farmclient.ani_x264farm.listener() in F:\[8]Coding\Ani-x264FARM Client\Ani-x264 (x264Farm) Client\Ani-x264Client.vb:line 423
Message: Value cannot be null.
Parameter name: path

---------------------------------------------------------------------

Listening for server connection

Did I miss a config step somewhere? I followed your guide tho :)
Also, I got an error message when I want to minimize the window on the client's machine.
************** Exception Text **************
System.IO.FileNotFoundException: Could not find file 'C:\Documents and Settings\Administrator\My Documents\Ani-x264FARMClient2000\Ani-x264 (x264Farm) Client.ico'.
File name: 'C:\Documents and Settings\Administrator\My Documents\Ani-x264FARMClient2000\Ani-x264 (x264Farm) Client.ico'
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
at System.Drawing.Icon..ctor(String fileName, Int32 width, Int32 height)
at System.Drawing.Icon..ctor(String fileName)
at ani_x264farmclient.ani_x264farm.OnResize(EventArgs e) in F:\[8]Coding\Ani-x264FARM Client\Ani-x264 (x264Farm) Client\Ani-x264Client.vb:line 562
at System.Windows.Forms.Control.OnSizeChanged(EventArgs e)
at System.Windows.Forms.Control.UpdateBounds(Int32 x, Int32 y, Int32 width, Int32 height, Int32 clientWidth, Int32 clientHeight)
at System.Windows.Forms.Control.UpdateBounds()
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ScrollableControl.WndProc(Message& m)
at System.Windows.Forms.ContainerControl.WndProc(Message& m)
at System.Windows.Forms.Form.WndProc(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

Thanks,
- Dan

kenji123

11th April 2008, 14:49

Date: #Apr.11.2008#
Time: [12:14 a.m.]

Stack Trace: at System.Windows.Forms.Control.MarshaledInvoke(Control caller, Delegate method, Object[] args, Boolean synchronous)
at System.Windows.Forms.Control.Invoke(Delegate method, Object[] args)
at ani_x264farmclient.ani_x264farm.listener() in F:\[8]Coding\Ani-x264FARM Client\Ani-x264 (x264Farm) Client\Ani-x264Client.vb:line 423
Message: Value cannot be null.
Parameter name: path

---------------------------------------------------------------------

Listening for server connection

Did I miss a config step somewhere? I followed your guide tho :)

No you didn't.:) This one is my fault, a required directory wasn't created by the program.

Also, I got an error message when I want to minimize the window on the client's machine.
************** Exception Text **************
System.IO.FileNotFoundException: Could not find file 'C:\Documents and Settings\Administrator\My Documents\Ani-x264FARMClient2000\Ani-x264 (x264Farm) Client.ico'.
File name: 'C:\Documents and Settings\Administrator\My Documents\Ani-x264FARMClient2000\Ani-x264 (x264Farm) Client.ico'
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
at System.Drawing.Icon..ctor(String fileName, Int32 width, Int32 height)
at System.Drawing.Icon..ctor(String fileName)
at ani_x264farmclient.ani_x264farm.OnResize(EventArgs e) in F:\[8]Coding\Ani-x264FARM Client\Ani-x264 (x264Farm) Client\Ani-x264Client.vb:line 562
at System.Windows.Forms.Control.OnSizeChanged(EventArgs e)
at System.Windows.Forms.Control.UpdateBounds(Int32 x, Int32 y, Int32 width, Int32 height, Int32 clientWidth, Int32 clientHeight)
at System.Windows.Forms.Control.UpdateBounds()
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ScrollableControl.WndProc(Message& m)
at System.Windows.Forms.ContainerControl.WndProc(Message& m)
at System.Windows.Forms.Form.WndProc(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

Thanks,
- Dan

Again my fault. When the program minimizes, it loads an icon for the system tray, which i failed to include.:) This error would of also occurred with the controller program. (i've fixed it so that you don't need the icon file.)

anyways;
i've fixed the bugs, so you can replace the old exes with the ones in this file here (http://kenji123.uni.cc/ani-x264/Ani-x2641500FARM2000_exeonly.7z)

the links in my previous post have also be updated.

thanks for trying it out and providing feedback :)

Razorholt

11th April 2008, 15:37

Ok, I'm trying now... Always glad to help ;)

Razorholt

11th April 2008, 15:55

log start: #Apr.11.2008_9.45.a.m_Married_Life_1080p.mov#

---------------------------------------------------------------------
copying input file to tmpfiles directory

Source: C:\Videos\trailers\Married_Life_1080p.mov
Destination: C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile.mov
Buffer Size: 1,000,000 Bytes/s

---------------------------------------------------------------------
remuxing to a mkv container

command line:
-o "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile_vid.mkv" --engage allow_avc_in_vfw_mode "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile.mov"

lines returned by the mkvmerge cli:

mkvmerge v2.0.2 ('You're My Flame') built on Feb 21 2007 23:40:43
'C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile.mov': Using the Quicktime/MP4 demultiplexer.
'C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile.mov' track 1: Using the MPEG-4 part 10 (AVC) video output module.
'C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile.mov' track 2: Using the AAC output module.
The file 'C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile_vid.mkv' has been opened for writing.
The cue entries (the index) are being written...
Muxing took 5 seconds.

---------------------------------------------------------------------
creating avisynth script

script generated:

LoadPlugin("F:\[8]Coding\Ani-x264\Ani-x264\bin\Debug\avsplugins\Undot.dll")
LoadPlugin("F:\[8]Coding\Ani-x264\Ani-x264\bin\Debug\avsplugins\FluxSmooth.dll")
LoadPlugin("F:\[8]Coding\Ani-x264\Ani-x264\bin\Debug\avsplugins\Convolution3DYV12.dll")
LoadPlugin("F:\[8]Coding\Ani-x264\Ani-x264\bin\Debug\avsplugins\AutoCrop.dll")
LoadPlugin("F:\[8]Coding\Ani-x264\Ani-x264\bin\Debug\avsplugins\Decomb.dll")
LoadPlugin("F:\[8]Coding\Ani-x264\Ani-x264\bin\Debug\avsplugins\VSFilter.dll")

function logo (clip orig, string image, int start, int length, int fadein, int fadeout, int trans, int x, int y, int threshold)
{
trans = Default(trans, 0)
fadein = Default(fadein, 0)
fadeout = Default(fadeout, 0)
x = Default(x, 0)
y = Default(y, 0)
threshold = Default(threshold, 0)
framerate = orig.framerate()
overlay = ImageReader(image, 0, length-1, framerate, false)
mask= overlay.ConvertToRGB32().ColorKeyMask(trans, threshold).ShowAlpha().FadeIO(fadein,fadeout)
numframes = overlay.framecount()
origclip = orig.Trim(start, start+length-1)
newclip = Overlay(origclip,overlay,x=x, y=y, mask=mask)
return orig.Trim(0, start - 1) + newclip + orig.Trim(start + numframes, 0)
}

DirectShowSource("C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile_vid.mkv",convertfps=true,audio=false) # Source
Lanczos4Resize(704,400) # Lanczos (Sharp)

logo("F:\[6]Graphics\My Graphics\otakuleg_textlogov1 copy.png",0,300,25,25,0,5,5,0)

Telecide() # Inverse Telecine
Decimate(5) # Inverse Telecine

FadeIO2(25) # FadeIn/FadeOut Frames
ConvertToYV12()

---------------------------------------------------------------------
connecting to clients across the network

lines returned by the x264Farm TCP Controller:

Connecting to client no. 1
Connection established with client no. 1
Sending job info to client no. 1
Job info finished sending; sending agent configuration
Sending agent configuration to client no. 1
Configuration finished sending; starting x264Farm agent
x264Farm agent is running on client no. 1
Disconnected from client no. 1
Working...
---------------------------------------------------------------------
starting x264Farm controller

command line:
--first "--qp 26 --bframes 16 --b-pyramid --subme 1 --analyse none --me dia --progress" --second "--merange 12 --me umh --subme 6 --filter 1,1 --trellis 1 --ref 8 --mixed-refs --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --bframes 16 --b-pyramid --b-rdo --weightb --bime --direct auto --progress" --bitrate 280kbps --avs "InputFile.avs" --output "OutputFile.mkv" --config "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile_controller_config.xml"

lines returned by the x264Farm Controller, Agent(s) and x264 cli:

x264farm agent version 1.15-182
x264farm controller version 1.15-182
Using config file "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile_agent_config.xml"
Using config file "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile_controller_config.xml"
Temp dir: "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186"
CONFIG:
Ports: 6060-6070
temp_dir: "C:\\Documents and Settings\\encoderman\\Desktop\\Ani-x2641500\\tmpfiles\\MarriedLifemov1182793186"
x264: "x264.exe"
agent list:
nice: 10
Local = 127.0.0.1:6060 (1,1)
agents: (1,1)
bobby4 = 192.168.0.1:6060 (1,2)
agent bases: [None]
pipe: false
compressions: 0001 - 0001
deleting old files: after 604800 seconds
UDP ping port: 40700
UDP listening port: 40701
UDP name: "Local"
trying port 6060
Removing stale files from temp dir...
bound to port 6060
listening...
working!
Listener set up
WARNING: avs2yuv exited with error 1; avs2yuv responded:

x264Farm did not end as expected
Trying to recoverx264farm controller version 1.15-182
Using config file "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile_controller_config.xml"
CONFIG:
temp_dir: "C:\\Documents and Settings\\encoderman\\Desktop\\Ani-x2641500\\tmpfiles\\MarriedLifemov1182793186"
agent list:
Local = 127.0.0.1:6060 (1,1)
bobby4 = 192.168.0.1:6060 (1,2)
WARNING: avs2yuv exited with error 1; avs2yuv responded:

---------------------------------------------------------------------
connecting to clients across the network

lines returned by the x264Farm TCP Controller:
Connection established with client no. 1
Connecting to client no. 1
Stopping x264Farm agent on client no. 1
x264Farm agent has been stopped on client no. 1
Disconnecting from client no. 1
Disconnected from client no. 1
Working...

---------------------------------------------------------------------

log end: #Apr.11.2008_9.46.a.m_Married_Life_1080p.mov#

I also noticed that I can't import AVS files. Is it correct?

kenji123

11th April 2008, 16:19

You will need to select or create a new profile. the profile it is set at isusing a custom AVS script i was using for personal usage. you can easily create or edit profiles by going to the settings tab.

It checks the script first for errors, also any files used in the script must have an absolute path (eg. not "AVISource("test.avi")" but "AVISource("D:\folder\test.avi")").

If thats not the case, can you post the AVS script?

Razorholt

11th April 2008, 17:27

I have updated my avs file. It seems to be working at that level but then, I got this:
Recent errors:
4:07 bobby4 failed with "End_of_file"
173 recv thread got exception; killing listener thread
4:09 bobby4 failed with "End_of_file"
172 first pass frames 0 to 3432
4:11 Local failed with "End_of_file"
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 SSE3 Cache64
172 controller-based encoding
4:13 Local failed with "End_of_file"
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 SSE3 Cache64
172 doing the following:
nice -n 10 "x264.exe" --qp 26 --bframes 16 --b-pyramid --subme 1 --analyse none --me dia --progress --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 2997/125 --pass 1 --stats "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov915614466\stats 6702cb.txt" -o NUL - 528x288
Last updated: 4:15
Got a connection from 192.168.0.150:1311
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
working!
Agent | FPS | Description
174 recv thread got exception; killing listener thread
Local | 0.00 | Sending [0-3432]
Thread received exception "End_of_file"
bobby4 | 0.00 | Waiting for job
closing socket
Local | 0.00 | Connected to 192.168.0.150:6060
socket closed
bobby4 | 0.00 | Disconnected
Thread received exception "End_of_file"
Recent errors:
closing socket
4:07 bobby4 failed with "End_of_file"
socket closed
4:09 bobby4 failed with "End_of_file"
4:11 Local failed with "End_of_file"
4:13 Local failed with "End_of_file"
Thread received exception ("An existing connection was forcibly closed by the remote host.\013\n",recv,)
Last updated: 4:15
closing socket
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
socket closed
Agent | FPS | Description

---------------------------------------------------------------------
connecting to clients across the network

lines returned by the x264Farm TCP Controller:
Connection established with client no. 1
Connecting to client no. 1
Stopping x264Farm agent on client no. 1
x264Farm agent has been stopped on client no. 1
Disconnecting from client no. 1
Disconnected from client no. 1
Working...

---------------------------------------------------------------------

log end: #Apr.11.2008_11.24.a.m_Married_Life_1080p.mov#

gnzrg25

11th April 2008, 18:34

Hi. Im newbie...
I'm trying to compile x264 with visual c++ 2005.
Get these errors.

1>------ Build started: Project: x264, Configuration: Release Win32 ------
1>Linking...
1>x264.obj : error LNK2019: unresolved external symbol _strncasecmp referenced in function _Parse
1>libx264.lib(encoder.obj) : error LNK2019: unresolved external symbol _x264_cqm_init referenced in function _x264_encoder_open
1>libx264.lib(encoder.obj) : error LNK2019: unresolved external symbol _x264_cqm_parse_file referenced in function _x264_encoder_open
1>bin/x264.exe : fatal error LNK1120: 3 unresolved externals
1>Build log was saved at "file://c:\Documents and Settings\Administrator\Desktop\X264_rev270_src\x264\build\win32\obj\x264_Release\BuildLog.htm"
1>x264 - 4 error(s), 0 warning(s)
========== Build: 0 succeeded, 1 failed, 1 up-to-date, 0 skipped ==========

tnx..

Razorholt

11th April 2008, 19:59

My customized AVS:
DirectShowSource("{source}",convertfps=true,audio=false) # Source
Lanczos4Resize(704,400) # Lanczos (Sharp)
ConvertToYV12()

Log:
starting x264Farm controller

command line:
--first "--qp 26 --bframes 16 --b-pyramid --subme 1 --analyse none --me dia --progress" --second "--merange 12 --me umh --subme 6 --filter 1,1 --trellis 1 --ref 8 --mixed-refs --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --bframes 16 --b-pyramid --b-rdo --weightb --bime --direct auto --progress" --bitrate 280kbps --avs "InputFile.avs" --output "OutputFile.mkv" --config "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\test2avi2147434371\InputFile_controller_config.xml"

lines returned by the x264Farm Controller, Agent(s) and x264 cli:

x264farm agent version 1.15-182
Using config file "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\test2avi2147434371\InputFile_agent_config.xml"
Temp dir: "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\test2avi2147434371"
Ports: 6060-6065
x264: "x264.exe"
nice: 10
agents: (1,1)
agent bases: [None]
pipe: false
compressions: 0001 - 0001
deleting old files: after 604800 seconds
UDP ping port: 40700
x264farm controller version 1.15-182
UDP listening port: 40701
Using config file "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\test2avi2147434371\InputFile_controller_config.xml"
UDP name: "Local"
CONFIG:
trying port 6060
temp_dir: "C:\\Documents and Settings\\encoderman\\Desktop\\Ani-x2641500\\tmpfiles\\test2avi2147434371"
Removing stale files from temp dir...
agent list:
bound to port 6060
Local = 127.0.0.1:6060 (1,1)
listening...
bobby4 = 192.168.0.1:6060 (1,2)
Listener set up
working!

I see Job 1 - analyzing - [working...] at the status bar and that's it. Nothing is happening :(

RickA

11th April 2008, 21:52

To those not able to follow the thread where I was talking about the nice GUI for x264farm created by kenji123 - thanks again btw. Um, the one that he posted right above mine... Which I was referring to in my post giving the option of using the MKV container with AAC, but no option for AC3 audio in the MKV container.

Then gave a good reason why it would be nice of him to implement said option. Since just about every DVD and Hi-Def disc under the sun has AC3 audio by default. As well as the vast majority of home sound systems having the support for directly decoding AC3 Dolby Digital more so than AAC.

I know x264 is only for encoding video content. How many of you seriously watch / encode video without putting an audio in the container with it to listen to? That is why I was addressing the GUI. Have you even tried his GUI to have an idea of what I was talking about before jumping in with comments?

Cheers,
R

[P]ako

12th April 2008, 23:35

I know x264 is only for encoding video content. How many of you seriously watch / encode video without putting an audio in the container with it to listen to? That is why I was addressing the GUI. Have you even tried his GUI to have an idea of what I was talking about before jumping in with comments?

Cheers,
R

Please read your post carefully, in your addendum you mentioned x264farm and AC3, not the GUI for x264farm and AC3. And because you were speaking of x264farm I assumed you were addressing the thread poster (Omion). There is no reason to get pissy over the internet when someone seems not to understand you.

SpAwN_gUy

14th April 2008, 15:15

Hi. Im newbie...
I'm trying to compile x264 with visual c++ 2005.
Get these errors.
1st) this should go to x264 compilation thread..
2nd) I've just mastered it (again.. yes.. za sikret tikniq, 2 build x264 under Win MSVS 2005 ... even with pathces,.. and gpac and pthreads.. bwahahahaha)
ok.. i'll help..
1) convert .sln from 2003version to 2005 (as you did, as i can see)
2) add to project: "cabac-a.asm" to Core/x86/ folder AND configure it to compile with nasm as OTHER .asm's

ok. here is compilation of Mine:
upd. removed.... due to producing lotsofsh_t...

SpAwN_gUy

15th April 2008, 15:08

ok.. i've made another one... this one is a bit MORE proper :) ..

x264 rev.819, modified, MSVC2005 build
General thread:
http://forum.doom9.org/showthread.php?t=130364

x264.gaussian.cplxblur.01.diff
Dark Shikari: - gaussian cplxblur: gives a tiny improvement in 2pass ratecontrol
x264_me-prepass_DeathTheSheep.01.diff
http://forum.doom9.org/showthread.php?p=1093523
x264_2pass_vbv.7.diff
http://thread.gmane.org/gmane.comp.video.x264.devel/3093/focus=3748
x264_hrd_pulldown.04_interlace.diff
- HRD and pulldown for HD compatibility, updated patch for interlacing
http://forum.doom9.org/showthread.php?p=1047919#post1047919

grab it here (http://rapidshare.com/files/107703602/x264.819.msvc2005.modified.exe)

nvubert

22nd April 2008, 16:35

your x264farm is what i needed for some long time for my HD encodes. I'm using it about 5 months and it's really a piece of good job.
Here's my setup:
Controller:Q6600@4.0GHz (wc) 4GBDDR2-1066 2xRaptor36GB raid0 and 2xF1 320GB raid0, selfmodded XPSP2
Agents:my HTPC with E6600@3.2GHz 2GB-DDR2-800, E4300@3.2FGHz 2GB-DDR2-800 and 3xA64 X2@2.66GHz 2GBDDR2-800 ( network 1Gbps )

My question is:
Is possible to configure controller that only second pass will be processed over the agents and first pass only on my local Q6600? ( i see that v1.15 have recognize local and auto configure with agent encoding that not use av2yuv.exe )
The reason is that on 720p encodes at first pass with no avs filters speed is very slow 8-30fps, the same source on megui on local only i have 20-50fps ( depending on source )
Maybe is some trick that allow encode only 2pass on x264farm with first pass encoded on standalone x264.exe
Thanks for help

SpAwN_gUy

22nd April 2008, 16:51

The reason is that on 720p encodes at first pass with no avs filters speed is very slow 8-30fps, the same source on megui on local only i have 20-50fps ( depending on source ) you are doing controller-based or agent-based encodings?
Maybe is some trick that allow encode only 2pass on x264farm with first pass encoded on standalone x264.exe there is a trick.
you make first-pass on standalone, then start x264farm.. and after it starts doing first pass.... quit it (terminate). then locate folder (i.e. "193949954353 your.avs") in controller's temp folder, then (i guess.. i haven't done it myself) delete dir (in the dir metioned before) named like "1-34534...." and replace(or create...) file "1-BC95ECB7B64E98DEB......txt" with your .stats file form first-pass standalone encode...

but, doing agent-based is alot faster(on my side here)

nvubert

22nd April 2008, 17:11

I do controller based encoding because i can't configure every machine for share network,avs filters,etc. However since v1.15 agent on my controller self-configure for agent based encoding ( detecting avs file and says agent encoding - there's no avs2yuv.exe in taskmanager for it ) and other computers doing controller-based.

This "trick" I already knew, but i need some automate this ( if this is only solution )
It is very complicate to do this "hack" on every encode

SpAwN_gUy

23rd April 2008, 08:12

I do controller based encoding because i can't configure every machine for share network,avs filters,etc. i can... teach you how to setup agent-based... (well, there is a tutorial for mpeg2 encoding in my signature)... and i can explain everything... 'cause now i do AVC reEncodings agent-based...

nvubert

23rd April 2008, 10:40

There's no need to teach things i already know :)

Problem is that some computers can't host avs filters ( sometimes i use lots of them and scripts are heavy ) because they are HTPC's for TV 24/7 and others media in houses my neighbors, but anyway thanks for good intention :thanks:

[P]ako

27th April 2008, 19:39

Omion, would it be possible to have a new build of x264farm that includes the AQ patch (already in cvs) and the experimental film grain optimization patch?

FGO patch: http://forum.doom9.org/showthread.php?t=137117&highlight=patch

SpAwN_gUy

2nd May 2008, 13:31

ako;1129577']Omion, would it be possible to have a new build of x264farm that includes the AQ patch (already in cvs) and the experimental film grain optimization patch?

FGO patch: http://forum.doom9.org/showthread.php?t=137117&highlight=patchthere is no reason to reCompile x264farm... just replace x264.exe itself.

if you wish.. i've made r839 MSVC2005 build of x264... (but without patches)... and i'm struggling with git-apply ('cause it would be much more simplyer.. then to applay them by hands..(as i did before..))

[P]ako

2nd May 2008, 15:41

there is no reason to reCompile x264farm... just replace x264.exe itself.

Did I miss something? If I remember correctly Omion said that x264farm works only with his MSVS build of x264.

Dustin013

2nd May 2008, 20:27

I have been playing around with x264 and I am loving the codec. I have found a nice setup to encode all my Divx backups to work on my PC and IPhone which is amazing. However, I am using dual Intel 3.2ghz Dual Core Xeon processors and I can still only encode around a 3:1 ratio (about 1/3 the play time length to encode a file) when encoding AVI into a x264 format in a MP4 wrapper.

x264Farm seems like the perfect solution to speed this whole process up. However, I am converting AVIs encoded in Xvid mainly and some Divx and it would seem x264farm doesn't support AVI encoding?

Is it possible to modify x264farm to allow encoding of AVI files and possibly setup a way to batch encode?

I currently have access to some servers at work I was given permission to test on. I have 11 servers I can use to try this out. The controller server is a mammoth with 16GB ram and top of the line dual core processors. Then I have 10 dell 1U units that have 2GB ram in them each and dual 2.8ghz DC Xeon processors. If I could get the big server to work as the controller and the other 10 as agents I think I could encode my entire 3TB collection of backups into Mp4 format fairly quickly.

Any suggestions?

driftr

4th May 2008, 20:21

Dustin013: you can convert any file to x264. I've done mpeg2, avi (xvid, divx, wmv), etc etc. with x264rfarm. x264farm is just a program that utilizes the x264 codec so there's virtually no difference other than the supported build of the codec.

You just have to feed x264farm an AVS script. If you haven't used avisynth that's what you need to read up on and there are many guides at doom9 to help with that.

And one thing to keep in mind for the controller/agent computers is you'll have to install avisynth and any codecs on the computers. So you might want to be sure the powers that be are aware the process isn't a light weight application (thou it shouldn't interfere with any usual processes).

SpAwN_gUy

7th May 2008, 08:27

ako;1133563']Did I miss something? If I remember correctly Omion said that x264farm works only with his MSVS build of x264. one page back there is rev819..
i have unpatched 839... (i can post, and upgrade to current rev.)
i have linux and will try to use it for patching sources.. 'cause my conditions do not allow me to set up git properly..

[P]ako

7th May 2008, 16:05

I see. When I have time, I will try with Shikari's build of x264 with the FGO patch.

Humbula

25th May 2008, 21:36

elo,

can you tell me how you calculate the bitrate needed for a part of the second pass? is it possible to calculate is from the passfile?

Adub

26th May 2008, 02:47

Use a bitrate calculator. Just punch it into google, use MeGUI, or any other variety of programs. Again, if you google it, you'll find hundreds. And this is probably the wrong thread.

Humbula

26th May 2008, 05:22

Use a bitrate calculator. Just punch it into google, use MeGUI, or any other variety of programs. Again, if you google it, you'll find hundreds. And this is probably the wrong thread.

No, you didn't understand my posting. Let me try again: x264 farm "cuts" the movie into parts, which will be encoded by the next machine, requesting a job. Lets say one of this parts has about 250 frames.
How does x264 farm calculate the bitrate (needed) for this part?

Adub

26th May 2008, 05:54

Humbula

26th May 2008, 06:15

The bitrate shouldn't change. When you specify a bitrate in the commandline, you are saying to the encoder "aim for this". It doesn't matter if you split up the file or not, the bitrate will stay the same, as you want uniform quality throughout your encode.

x264 farm's aim in NOT to keep the bitrate for every part the same.

"The ratecontrol used is designed to be as close to the ratecontrol done by x264 itself"

I want to know how this ratecontrol works...

Adub

26th May 2008, 20:35

Oh, okay. I am sorry, I didn't fully comprehend what you are asking. And frankly, I don't know the answer. Probably the only way to find out is to look at the source or talk to Omion.

Humbula

26th May 2008, 21:32

to be honest i dont understand the code. strange language... :)

omion, help me...

foxyshadis

27th May 2008, 00:17

x264farm just processes the first pass stats files to come up with each segment's relative sizes.

Humbula

27th May 2008, 06:44

x264farm just processes the first pass stats files to come up with each segment's relative sizes.

And how can I calculate a segments size using the statsfile?

omion

27th May 2008, 06:49

to be honest i dont understand the code. strange language... :)

omion, help me...
The language is strange, but my coding style makes it even worse. You are very brave for attempting to read it :D

And how can I calculate a segments size using the statsfile?
Basically, the number of bits used by each frame is itex+ptex+mv+misc. Divide by 8 to get bytes (it should be a whole number of bytes)

[edit]If you want to know how the ratecontrol works, start by looking at x264's encoder/ratecontrol.c file. That's where I got my info. Using x264farm's source will only get you more confused. :)

Humbula

27th May 2008, 23:00

The language is strange, but my coding style makes it even worse. You are very brave for attempting to read it :D

:goodpost:

Basically, the number of bits used by each frame is itex+ptex+mv+misc. Divide by 8 to get bytes (it should be a whole number of bytes)

[edit]If you want to know how the ratecontrol works, start by looking at x264's encoder/ratecontrol.c file. That's where I got my info. Using x264farm's source will only get you more confused. :)

Thanks, I'll have a look :)

akupenguin

28th May 2008, 02:40

Basically, the number of bits used by each frame is itex+ptex+mv+misc. Divide by 8 to get bytes (it should be a whole number of bytes)
Before trying this at home, note that it only works if the 1st pass is CRF. x264's own bit-allocation algorithm gives similar results but isn't so picky.

omion

28th May 2008, 06:05

Before trying this at home, note that it only works if the 1st pass is CRF. x264's own bit-allocation algorithm gives similar results but isn't so picky.

Ah. Yes. I just realized that I actually answered a different question than what was asked. The size of the first pass frames is in itex+ptex+mv+misc (which was the question I was answering), but that does not directly correspond to the second pass frame size (which was the question asked). The stats file is processed through the functions in ratecontrol.c before the optimal second pass frame sizes are known.

Even after processing by the ratecontrol functions, there is no guarantee that the frames will actually be that size. In fact, since x264 can't directly set the number of bits used in a frame, it will always be a little off.

taemun

28th May 2008, 11:42

Adub

28th May 2008, 19:56

6> Given that x264 is highly multi-threaded is there any reason to have more than one instance of x264 running on every agent.

I was wondering about this too. Wouldn't it be faster to just let x264 run multithreaded?

foxyshadis

28th May 2008, 23:27

I was wondering about this too. Wouldn't it be faster to just let x264 run multithreaded?

Depends on if the input is bottlenecked or not. (Combine MT+x264 threading and you get better results.)

On the other hand, why does it need so much ram at all?! Graphedit can play back the full source in question with 110MB of RAM usage, why should a frame-server need any more than that?

Avisynth caches. You can use setmemorymax to control that, since avisynth caching ignores multiple instances. (This is even worse on 2.58, where default is half the available system memory, not one quarter. If they all initialize at once, they all see the same available memory.) Even if frames are never needed again, they'll sit in cache until their number comes up when the maximum memory limit is reached. Then it just stays at that limit until you're done encoding, mostly.

omion

28th May 2008, 23:49

Hey all.

I'm new to x264 encoding in general, (have worked out what most of the buttons in MeGUI etc do), and given that I have several computers lying around here, I thought that there should be a relatively easy way to get x264 to run over all of them at once.

Hence, found this project.

I've had it running, although I'd have to admit that a few things aren't clear to me:

1> Unless I specify a rate control method in the -1 "blah blah", it doesn't do anything at all.... was this intended? What value should I be using in here for a good quality 720p rip (is --qp 26 sufficient?).
That is an x264 issue. If you don't specify the ratecontrol method with x264, it will just fail with a helpful warning. The problem with x264farm is that the helpful warning isn't actually displayed...

I generally use a lower setting (--qp 22 or something) but do a few tests to see what looks good for you.

2> I'd like a little more control over the IP that the controller and agents bind to ... I have multiple IP addresses, that will go differing routes to get to the clients. One of those interfaces is gig, one is 100M. I'd rather be able to force it to go the gig path. This may be related to my inability to work out the networking parameters in the controller - the first time I tried I couldn't get anything aside from adhoc working - this suits my purposes fine, as you can kill the processes remotely, but adhoc can't specify the controller IP to be bound to.

Currently it's set to bind to any IP address, since I didn't need the ability to change it. That feature would be nice for some people, though, so I'll add it in. However, it's probably not going to get into the 2-pass version of x264farm any time soon, as I'm mainly working on the 1-pass version now.

3> I'd like to be able to change the agents list/the number of x264 instances per agent whilst the controller is running. You can sort of do this by running in adhoc, and just starting/closing/restarting the agent whenever you want to change something, but that only works (for me) if you can force the IP that the adhoc controller is running on.
I have (pretty buggy) support for that in the 1-pass version. It does still need you to restart the agent every time you want to change the settings, but it generally works. In non-adhoc mode for the 2-pass version, the agent list is defined immutably in the beginning of the encode. Changing the way that works would be fairly difficult.

4> What is AVS2YUV doing here? I presently have three instances of x264 running over two computers (one on controller, ~500MB/ram, two remote, ~230MB/ram each), and the controller also has three avs2yuv's running:
One of 766MB (no CPU-time, unless an agent recently finished a block).
Two of 550MB (3~13% CPU-time). Presumably these are frameserving across the network (controller based encoding).
[as I've been typing this, both of the 550MB processes have died, and two new ones of around 350MB have come up]

Reason I don't understand, is because x264 can handle an .avs input directly, why is there a need for a frame server here? (proviso: during agent-based encoding)
On the other hand, why does it need so much ram at all?! Graphedit can play back the full source in question with 110MB of RAM usage, why should a frame-server need any more than that?

Any controller-based encode needs to start up its own instance of avs2yuv to render the AVS file. There is also one extra avs2yuv which the controller consults for finding scene cuts. The memory usage is always a bit large by default (since Avisynth is set up to cache everything it gets its hands on) but you can limit that by using SetMemoryMax(...) in the AVS file. You are correct that frameserving should use very little RAM - every output frame is used exactly once, and in order. Therefore you should see no performance decrease by limiting the amount of memory Avisynth uses.

5> I can't seem to get agent-based encoding working, the agent throws the error:
Presumably, the AVS component is working (it is AVS throwing an error about the .grf file), and graphedit maybe doesn't like working across a Windows File Share...? U:\ is just mapped to \\10.0.0.1\u$

I can't really help you there. I don't know the inner workings of graphedit or MeGUI... Generally errors like that are due to some path being hard-coded in the AVS (or an included file), but since you have the agent's U:\ mapped to the controller's U:\, I don't really know.

6> Given that x264 is highly multi-threaded is there any reason to have more than one instance of x264 running on every agent. Aside from the small delay that occurs when dumping back the stats file, or the resulting .mkv files in the second pass. IF you're worried about that delay in the second pass, I can see why you might use pad=1; number=1. (btw, why not simplify that into, numberFirstPass=1; numberSecondPass=2?)
I honestly don't know how they compare. When x264 first added threading support I tested it out, and using multiple single-threaded x264 processes worked a little faster. But then the threading model was completely changed around and it looks like it's quite a bit better now than it used to be. Test it out and report back! (oh, and use "--threads auto", or it will force all agents to use the same number of threads, which is probably not what you want;))

foxyshadis

29th May 2008, 06:25

5> I can't seem to get agent-based encoding working, the agent throws the error:
Presumably, the AVS component is working (it is AVS throwing an error about the .grf file), and graphedit maybe doesn't like working across a Windows File Share...? U:\ is just mapped to \\10.0.0.1\u$
Come to think about it, this is probably because you don't have all of the exact same directshow filters registered on every agent as the controller. Filters are specified by CLSID in the .grf, so even different versions of the same filter may not be picked up.

taemun

29th May 2008, 07:02

Come to think about it, this is probably because you don't have all of the exact same directshow filters registered on every agent as the controller. Filters are specified by CLSID in the .grf, so even different versions of the same filter may not be picked up.

:D
Yeah I came to that conclusion late last night. CoreAVC was only installed on the controller, so of course it failed. Pity I didn't think about that at the time (after trying to dig through the .grf with notepad :P). I'd forgotten that I had used something a little more exotic when I made the graph.

That is an x264 issue. If you don't specify the ratecontrol method with x264, it will just fail with a helpful warning. The problem with x264farm is that the helpful warning isn't actually displayed...

Yeah I did see that helpful warning, was more wondering whether I was doing something wrong, because the FAQ (http://omion.dyndns.org/x264farm/x264farm.html) didn't mention it.

That feature would be nice for some people, though, so I'll add it in.

Thanks, that'd be great.

There is also one extra avs2yuv which the controller consults for finding scene cuts.

That would be the one that had 3MB in RAM and ~750MB in pagefile at the end :)

Not to tell you your job or anything, but shouldn't that instance be able to be closed after the first pass is complete? (ie after all the scene cuts have been found)

On another note, why is it that the x264 builds used by x264farm have to be made in MSVC? (ie not from x264.nl)

And: what is the newest MSVC build. The one posted a couple pages back by SpAwN_gUy called x264.819.msvc2005.modified.exe, seems to crash quite a bit (I think its when x264 has too low of a bitrate, and normally it would throw an error ... but instead, this build just crashes and then Windows sits and waits for user input - ie to close the "this program has crashed" dialog).

Cheers,
t

omion

29th May 2008, 07:43

That would be the one that had 3MB in RAM and ~750MB in pagefile at the end :)

Not to tell you your job or anything, but shouldn't that instance be able to be closed after the first pass is complete? (ie after all the scene cuts have been found)

That would certainly make the most sense, wouldn't it?:p The only real reason I didn't do that is that I had abstracted away the avs2yuv process so far that it hid all the parts that I could have closed. Rather than attempting to do that, I decided instead to try and get rid of the splitter altogether. After much hacking away, I finally decided to do a ground-up rewrite to the entire program, which turned into x264farm-sp (which doesn't have a splitter). The 2-pass x264farm has since fallen by the wayside.

On another note, why is it that the x264 builds used by x264farm have to be made in MSVC? (ie not from x264.nl)

That is due to an annoying bug that x264 will open the stdin handle in text mode, meaning that it will interpret ASCII character 0x1A as an EOF marker and stop encoding. (it also turns 0x0D0A into 0x0D) MSVC builds don't have this issue, though, as there are a few MSVC-specific lines in x264 to switch to binary.

This is only a problem in controller-based mode, though. You may use anything you want if you ensure that the computer is using agent-based encoding.

(Note that it's not really a bug in x264; I think stdin and stdout are usually opened in text-mode in Windows. It's just a bit odd that x264 expects binary data to pass through them...)

It should be an easy fix for making it work in cygwin/msys, but I really don't know enough C to say for sure.

And: what is the newest MSVC build. The one posted a couple pages back by SpAwN_gUy called x264.819.msvc2005.modified.exe, seems to crash quite a bit (I think its when x264 has too low of a bitrate, and normally it would throw an error ... but instead, this build just crashes and then Windows sits and waits for user input - ie to close the "this program has crashed" dialog).

Cheers,
t
I'm not really sure what the most current version of MSVC x264 is. I gave up making my own builds a long time ago, since x264 is really only tested against GCC, and some of the changes would break MSVC for long periods of time. Not knowing a lot of C, I was clueless to fix it in these cases. Since then, I have had no need for controller-based encoding (on Windows boxes), so I haven't kept up with the progress. I'll have to defer to SpAwN_gUy to answer that question.

If you can get agent-based encoding up and running, then you can bypass this restriction. That would probably be the best course of action right now. (if the CoreAVC fix made agent-based encoding work, then you can use anything you want)

SpAwN_gUy

2nd June 2008, 11:58

i've just moved to the new PCbox ... so i've lost the tracks of the threads...

okay, since b0bor started to post patched sources (damn git..) it will not be too hard to make MSVC builds ...

good news are: meGUI-x264farm-mod (ma-a-n... this is quite a huge name...) is still indev.. i've moved 80% of code to the latest (like two weeks ago) meGUI revision... so.. some may wait for a new version soon..

/*offtop... as far as i'm learning bash and perl (and doing some fancy scripts..).. C# is my "back to the reality" thing... so more to a hobby.. but i'll catchUP :) ... and i'm on Vista x64 now..

Adub

2nd June 2008, 22:03

Good to hear from you again. As you know, looking forward to the mod!! Thanks again.

thebigfatgeek

7th June 2008, 23:19

Hi, I have just installed the agents and the controllers, modified th config.xml files and I keep on getting the same result. I cannot see the error, or that anything is happening (except some processor activity on controller.exe and avs2yuv, but after 0:46 seconds, it stops, it does not complain (which is nice) but it also does not work (which is not nice)

This is the output from the out-dump.txt file:

2008-06-07~23:09:23.53 0% done (0 / 49544) at 0.00 FPS
2008-06-07~23:09:23.53 Last updated: 0:45
2008-06-07~23:09:23.53 ----...----________________________________...............-----...............
2008-06-07~23:09:23.53 Agent | FPS | Description
2008-06-07~23:09:23.53 Recent errors:
2008-06-07~23:09:23.53 ~
2008-06-07~23:09:23.53 ~
2008-06-07~23:09:23.53 ~
2008-06-07~23:09:23.53 ~
2008-06-07~23:09:23.53 SPLITTER MUTEX unlocking range_list_mutex and yielding
2008-06-07~23:09:23.53 SPLITTER MUTEX sleeping for 1/10 second
2008-06-07~23:09:23.64 SPLITTER MUTEX locked range_list_mutex after yielding
2008-06-07~23:09:23.64 e 0
2008-06-07~23:09:23.64 f 0
2008-06-07~23:09:23.64 q 3
2008-06-07~23:09:23.64 w 0
2008-06-07~23:09:23.64 es 3
2008-06-07~23:09:23.64 fs 0
2008-06-07~23:09:23.64 qs 8
2008-06-07~23:09:23.64 ws 0
2008-06-07~23:09:23.64 Splitter: current batch length is 2116
2008-06-07~23:09:23.64 Splitter: try_queue_after_full_or_working
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 Splitter: try_extend_small_thresh_queue
2008-06-07~23:09:23.64 extend_small_thresh_queue found no ranges of sufficient patheticosity
2008-06-07~23:09:23.64 Splitter: try_split_largest_range
2008-06-07~23:09:23.64 Splitter: wait_on_full
2008-06-07~23:09:23.64 SPLITTER MUTEX broadcasting range_list_empty_condition
2008-06-07~23:09:23.64 0 @ INFINITY
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 2502 @ 7.56
2008-06-07~23:09:23.64 Empty
2008-06-07~23:09:23.64 4607 @ 10.63
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 6857 @ 4.69
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 9282 @ 2.17
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 11709 @ 7.17
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 14176 @ 2.42
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 16791 @ 6.33
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 19347 @ 7.58
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 22022 @ 9.34
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 24696 @ 4.79
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 27246 @ 3.65
2008-06-07~23:09:23.64 Empty
2008-06-07~23:09:23.64 37171 @ 11.85
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 39525 @ 6.16
2008-06-07~23:09:23.64 Empty
2008-06-07~23:09:23.64 49544 @ INFINITY
2008-06-07~23:09:23.64 0% done (0 / 49544) at 0.00 FPS
2008-06-07~23:09:23.64 Last updated: 0:45
2008-06-07~23:09:23.64 ----...----________________________________...............-----...............
2008-06-07~23:09:23.64 Agent | FPS | Description
2008-06-07~23:09:23.64 Recent errors:
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 SPLITTER MUTEX waiting on range_list_full_condition; unlocking range_list_mutex

and my controller was started with:

Controller>controller -B 4500kbps --first "--bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --me dia --me range 12 --progress --no-psnr" --second "--ref 3 --bframes 3 --b-pyramid --weigh tb --direct auto --filter -2,-1 --subme 6 --trellis 1 --analyse all --8x8dct --m e umh --merange 12 --progress --no-psnr" --avs "24.avs" -o "24.mkv" --batch 2500 --split 250 --preseek 5 --config config.xml

Any ideas or comments would be appreciated

Cheers

ajp_anton

16th July 2008, 15:33

Bug!

If the .avs file to encode is on a network share, it looks like "\\computer\folder"
When nice is used, it reads the "\\" as "\". It needs "\\\" to get it right, but then the .avs file won't be found and avs2yuv starts sending massive amounts of data across the network.

Apparently I can get around this by creating a .avs file that finds its inputs via the network, and copy that file onto every computer, even the local one (since it seems intelligent enough to check if the files are the same... =)

Otherwise this thing is really the best thing ever. Too bad it's not until now that I really needed it as it took only 10 mins to figure everything out and get it working.

SpAwN_gUy

17th July 2008, 09:08

If the .avs file to encode is on a network share, it looks like "\\computer\folder"
When nice is used, it reads the "\\" as "\". It needs "\\\" to get it right, but then the .avs file won't be found and avs2yuv starts sending massive amounts of data across the network. i have no problem with that ...
which nice are you using? (the one in the first post) jr the original? .. 'cause we had some problems using the original one.. and some problems using the tiny one(1st post)

currently i'm using the one in the first post... and i'm doing agent-based..

SpAwN_gUy

17th July 2008, 09:12

2008-06-07~23:09:23.64 Last updated: 0:45
2008-06-07~23:09:23.64 ----...----________________________________...............-----...............
2008-06-07~23:09:23.64 Agent | FPS | Description
2008-06-07~23:09:23.64 Recent errors:
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 SPLITTER MUTEX waiting on range_list_full_condition; unlocking range_list_mutex
controller haven't found any agents on the network..
check <ad-hoc> presence in controller's config..
AND try adding IP's of your agents like the old way (some of my agents are not seen some times with ad-hoc only.. but with both - it's perfect..)

SpAwN_gUy

17th July 2008, 09:18

oh.. jeah.. right.. the mod.. i need to start MSVC... and stop doing nothing...

btw.. about builds of x264... it is reported, that MSVC builds (with current revision of x264) are TOTALLY BROKEN.. and must not be used :(( ...
but the good nwes are.. that x264_mingw_stdin_patch is IN the TRUNK :) .. (yeah, yeah,.. thanks to me.. but you are welcome..) so all the builds, that are currently made.. (like on x264.nl and all the patched ones) need to be checked(at first).. and then.. well.. the must be working and compatible with controller-based x264farm encoding process..

thebigfatgeek

19th July 2008, 02:19

controller haven't found any agents on the network..
check <ad-hoc> presence in controller's config..
AND try adding IP's of your agents like the old way (some of my agents are not seen some times with ad-hoc only.. but with both - it's perfect..)

I can get it to see the agent, but I just cannot get a controller startup command that will work:

nice -n 20 "x264.exe" --bframes 3 --b-pyramid --direct auto --filter -2,-1 --s
bme 1 --analyse none --progress --no-psnr --qcomp 0.6 --cplxblur 20. --qblur 0.
--qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 25/1 --pass
1 --stats ".\temp\stats b26912.txt" -o NUL - 640x496
264 [error]: no ratecontrol method specified
264 [error]: x264_encoder_open failed

Wishbringer

19th July 2008, 10:44

Controller>controller -B 4500kbps --first "--bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --me dia --me range 12 --progress --no-psnr" --second "--ref 3 --bframes 3 --b-pyramid --weigh tb --direct auto --filter -2,-1 --subme 6 --trellis 1 --analyse all --8x8dct --m e umh --merange 12 --progress --no-psnr" --avs "24.avs" -o "24.mkv" --batch 2500 --split 250 --preseek 5 --config config.xml

You did set up the final bitrate for the second pass, but

264 [error]: no ratecontrol method specified

look at first pass string , there is no ratecontrol set. I would add something like --crf 18 into first pass string

see: http://forum.doom9.org/showpost.php?p=1143312&postcount=589 some posts before.

SpAwN_gUy

5th August 2008, 10:51

hi, guys, i've been asked to help with x264farm setup...
and i'm on the new pcBox ... so i had some time.. to just test everything from the scratch or something like this...
http://www.ljplus.ru/img4/s/p/spawn_guy/th_x264farm-vista-x64-doubleAgent-gcc_928.jpg
fullPicture is here (http://www.ljplus.ru/img4/s/p/spawn_guy/x264farm-vista-x64-doubleAgent-gcc_928.jpg)
the Picture shows win Vista SP1 x64 ... with two running agents.
one from commandline, - this one is doing controller-based
second started over the network through my agentGUI. - this one is agent-based
they both use nice at 20,
and mingw compiled x264 x86 r928 from b0bor's site .. (plain git)
no x264x64 (i havent found one x64 for now)
aviSynth 258.. but x86

test encode uses mpeg2-progressive source, mpeg2-Telecined with YATTA, x264-source form HDTVstream using DGAVCDec..

to get all thing working i needed to (this is imporatnt) set nice's compatibility mode to winXP-SP2 ... otherwise the whole thig was not working.. (but only for agents...)

and.. UNBLOK controller and agents in winFirewall !!!

and that's just it :) ...
any questions are welcome ....
(i guess, i should better show the meGUI-Mod.. but it is still unusable... (well, the new version))

controller's config.xml
<config>
<temp>controller-temp</temp>

<adhoc controller="40700" agent="40701"/>
<agents>
</agents>
</config>
agent's config (the command-line controller-based one):
<config>
<temp>temp</temp>
<port from="40700" to="40701"/>
<x264>x264.exe</x264>
<nice>20</nice>
<adhoc controller="40700" agent="40701"/>
<number>1</number>
<agentpipe>0</agentpipe>
<compression>
<type>1</type>
<type>0</type>
</compression>
</config>
angent's config (the one that is common and is started over the network with my agentGUI (it can be started the same way using "\\myPC\x264farm\!AGENT\agent.exe --config \\myPC\x264farm\!AGENT\config.xml")):
<config>

<temp>temp</temp>
<port from="40700" to="40701"/>
<x264>\\myPC\x264farm\!AGENT\x264.exe</x264>
<nice>20</nice>
<bases>
<base>\\myPC\x264farm-2\</base>
<base>\\myPC\x264farm\</base>
</bases>
<adhoc controller="40700" agent="40701"/>
<number>1</number>
<agentpipe>0</agentpipe>
<compression>
<type>1</type>
<type>0</type>
</compression>
</config>

theese are the internals of test.avs used:
SetMemoryMax(64)

# PLUGINS
LoadPlugin("\\myPC\x264farm\TEST\filters\DGDecode.dll")
LoadPlugin("\\myPC\x264farm\TEST\filters\FluxSmooth.dll")
LoadPlugin("\\myPC\x264farm\TEST\filters\DGAVCDecode.dll")
LoadPlugin("\\myPC\x264farm\!filters\TDeint.dll")
LoadPlugin("\\myPC\x264farm\!filters\TIVTC.dll")

function mpeg2(){
clip1=mpeg2source(".\mpeg2\test.d2v", idct=0)

clip1=crop(clip1,14,82,696,412)

return LanczosResize(clip1,640,480)
}

function mpeg2anime(){

d2vpath = ".\mpeg2-anime\yatta-test.d2v"
clip1=mpeg2source(d2vpath, idct=0)

deint = tdeint(clip1)
clip1 = tfm(clip1, d2v=d2vpath, clip2=deint)

return LanczosResize(clip1,640,480).AssumeFPS(25).Trim(400, 3400)
}

function ColBars() {
return ColorBars(640,480).AssumeFPS(25).ConvertToYV12().KillAudio.Trim(0,1000)
}

function h264() {
clip2=AVCSource(".\h264\hgttg-test.dga", deblock=false)

return clip2.AssumeFPS(25).ConvertToYV12().KillAudio.LanczosResize(640,480)
}

mpeg2().Info.Overlay(Version, x=200, y=400) + ColBars().Info.Overlay(Version, x=200, y=400) + h264().Info.Overlay(Version, x=200, y=400) + mpeg2anime().Info.Overlay(Version)

the is: E:\DVDRIP\x264farm\TEST\mpeg2\test.d2v internals (the .vob is located just near E:\DVDRIP\x264farm\TEST\mpeg2\VTS_01_8.VOB)
DGIndexProjectFile16
1
VTS_01_8.VOB

Stream_Type=1
MPEG_Type=2
iDCT_Algorithm=5
YUVRGB_Scale=1
Luminance_Filter=0,0
Clipping=0,0,0,0
Aspect_Ratio=16:9
Picture_Size=720x576
Field_Operation=0
Frame_Rate=25000 (25/1)
Location=0,0,0,b9de

900 5 0 20480 0 7 18 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 e2
d00 5 0 258048 0 7 18 d2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
.......
this is E:\DVDRIP\x264farm\TEST\h264\hgttg-test.dga
the only part i har to change manually is path to libavcodec (this is not implemented to be relative-path .. i had to change it to network-path)
DGAVCIndexFile5

hgttg-test.h264
\\myPC\x264farm\TEST\filters\libavcodec.dll

FO 2

LPOC 0
SPS 0
PPS 28

this is E:\DVDRIP\x264farm\TEST\mpeg2-anime\yatta-test.d2v (the one that is manually inverse-Telecined with YATTA ( ivtc.org ))
DGIndexProjectFile16
1
VTS_01_5.VOB

Stream_Type=1
MPEG_Type=2
iDCT_Algorithm=5
YUVRGB_Scale=1
Luminance_Filter=0,0
Clipping=0,0,0,0
Aspect_Ratio=4:3
Picture_Size=720x480
Field_Operation=0
Frame_Rate=29970 (30000/1001)
Location=0,0,0,44837

900 5 0 2048 0 5 2 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2 b2 b2 a2
...
aviSynth filters are located here
localPath: E:\DVDRIP\x264farm\TEST\filters\
networkPath: \\myPC\x264farm\TEST\filters
basically.. my setUP is to use network-share:
localPath: E:\DVDRIP\x264farm
networkPath: \\myPC\x264farm
for my agentGUI i use common agent setUP located here:
local: E:\DVDRIP\x264farm\!AGENT
network: \\myPC\x264farm\!AGENT

that's i folk :) ... it's pretty simple...

the some kind of minimal setUP can be obtained form here (http://rapidshare.com/files/134974974/x264farm-minimal.rar)

winAMP plays: [Apex - Same Old Blues]:[Drum & Bass-2008]:-:[5:40s]:[320-44KHz-stereo]:[MP3]

BTW: if you find any confidential data in things, that i wrote here .. please.. DON'T USE IT.. and tell me about it..
some mistakes can be.. that "myPC" real network name is "SC0151" remember that...

Jabroni

15th September 2008, 23:05

omion

15th September 2008, 23:24

Im trying to get this working with a newer X264 build (techouse builds), however Im getting some probs. I have been able to make it process the frames, the agent doesnt display any errors at all, neither on the controller, but the % of what is done never increases (it seems like the agent isnt returning the encoded info)

Any place what I should be looking into ??

:thanks:

I figured this may come up. x264 recently changed the stats file format, and I had an idea that would break x264farm somehow. Your description of the problem is exactly what I would expect to happen, so this is almost definitely the issue. The fix shouldn't be too hard, but I haven't built this version of x264farm in such a long time that I don't know how long it will take to start back up.

Jabroni

15th September 2008, 23:30

I figured this may come up. x264 recently changed the stats file format, and I had an idea that would break x264farm somehow. Your description of the problem is exactly what I would expect to happen, so this is almost definitely the issue. The fix shouldn't be too hard, but I haven't built this version of x264farm in such a long time that I don't know how long it will take to start back up.

Ok good.. thought this was just me being stupid on setting the new build :p Well for now i'll just stick with an old x264.exe build (the one that I had working previously)

This is the url of the custom build that I tried hxxp://techouse.project357.com/ (latest svn release) If you need any more info for debug let me know and I'll happly try to help :)

kemuri-_9

16th September 2008, 00:54

I figured this may come up. x264 recently changed the stats file format, and I had an idea that would break x264farm somehow. Your description of the problem is exactly what I would expect to happen, so this is almost definitely the issue. The fix shouldn't be too hard, but I haven't built this version of x264farm in such a long time that I don't know how long it will take to start back up.

if you really need to get back to the old stats format, you can just make a build that has that revision which affected it reversed.
keep all the other improvements and such...

SpAwN_gUy

16th September 2008, 14:42

so... we are back in bizzznisss? ;) ... cool :) ...

'cause re-encoding avchd from my sony hdr-sr12 to something reasonable at 720p... is really painful.. just "standalone"...
megui-mod update is really close :)

ajp_anton

28th September 2008, 19:29

So are we going to see a version of x264farm that supports the new stats files?

omion

29th September 2008, 16:57

So are we going to see a version of x264farm that supports the new stats files?

It'll be out eventually. The problem that I've had is that after I fixed the stats problem I noticed that the encode does not progress, so I'll have to fix that as well, and I haven't looked at the x264farm code in a long time. Part of the reason it's taken so long is that I also updated the regular-expression parser because I no longer have PCRE on my system. So it's still being worked on...

Nakaleen

8th October 2008, 02:22

Hi I am new to x264farm. Arfter some playing around I got x264farm to work over a network, also for it to work inside Staxrip.

My problem is that x264 keeps crashing on all my computers on the last section of the second pass. All it says is 'x264 as stopped working' and waits for me to close that error window, which is a pain when I leave it overnight only to find that is hasen't even finished one encode hehehe.

I have tried diffrent x264 builds, diffrent os (also crashes in the same spot on XP) a later version of Nice (ever tried to find the program 'Nice' via google, not easy hehehe) but x264 still reports an error

Here is an example of my setting (it is a modified version of Staxrip's Highprofile settings)

Controller.exe --first "--bitrate 907 --level 4.1 --bframes 3 --b-pyramid --threads auto --progress --sar 1:1 --no-psnr --no-ssim --subme 1 --partitions none --me dia" --second "--bitrate 907 --level 4.1 --ref 3 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --subme 6 --partitions all --8x8dct --trellis 1 --threads auto --progress --sar 1:1 --no-psnr --no-ssim" --output "R:\Movies\Ah My Goddess\Ah My Goddess_EncoderOutput.mkv" --avs "R:\Movies\Ah My Goddess\Ah My Goddess.avs"

I am running with Agent Encoding...

I think that all I can add, I've tried sifting through the out-dump.txt files but can't see anything that says what caused the errors.

I would LOVE for this to work as it has increased my encoding speeds from 12fps to about 30-35fps (if I keep an eye on all my computers to close down all the error boxes hehehe)

Sagekilla

8th October 2008, 02:28

Is there any chance you could actually use x264farm for crf 1-pass? I can imagine it wouldn't be much more complex, since it would be more or less identical to 1-pass.

professor_desty_nova

8th October 2008, 08:31

Is there any chance you could actually use x264farm for crf 1-pass? I can imagine it wouldn't be much more complex, since it would be more or less identical to 1-pass.

Are you talking of x264farm-sp - single-pass distributed encoding (http://forum.doom9.org/showthread.php?t=136053) ;)

SpAwN_gUy

8th October 2008, 09:21

My problem is that x264 keeps crashing on all my computers on the last section of the second pass. All it says is 'x264 as stopped working' and waits for me to close that error window, which is a pain when I leave it overnight only to find that is hasen't even finished one encode hehehe.how are you running agents?
from console or from "staxrip-agent"?

if from console the you could see x264's error.. if from satxrip - then you will not see the message...

i've had some "crashings" also... and (as i remember) i've implemented part of code to mine agentGUI to suppress those "XP-Fault-Windows" ... and i think i did the same thing to my delphi-nice-implementation (or not mine.. ).

but i still have "new-vista-error-windows" (haven't checked for new solution yet)

so.. or you kick staxrip-writer to implement "suppressors" (it's just 1 line of code on delphi) .. or you wait for new x264farm (i've told omin this issue.. long time ago).. or i'll look for my "nice.exe" somewhere here ...
"suppressors" must be implemented in a higher level application..
like staxrip-agent -then-> x264farm-agent -then-> nice.exe -then-> x264.exe

Nakaleen

11th October 2008, 10:34

Sorry for the delay. First week back from work so I haven't had time to play around with this again.

how are you running agents? from console or from "staxrip-agent"?

I don't know what you mean by 'Staxrip-agent'. Agent is running on every computer (3) from a console (started from a shortcut).

Each agent has the following 'config.xml' (execept for the <name> and <port> which is diffrent)

<config>
<temp>C:\Agent</temp>
<port from="40720" to="40729"/>
<adhoc controller="50700" agent="50701"/>
<name>Main</name>
<number>2</number>
<x264>x264.exe</x264>
<nice>10</nice>
<base></base>
<agentpipe>0</agentpipe>
<compression>
<type>0</type>
<type>1</type>
</compression>
</config>

I am using Staxrip with it's Commandline encoding option with the following....

C:\Controller\Controller.exe" --first "--bitrate %video_bitrate% --level 4.1 --bframes 3 --b-pyramid --threads auto --progress --sar 1:1 --no-psnr --no-ssim --subme 1 --partitions none --me dia" --second "--bitrate %video_bitrate% --level 4.1 --ref 3 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --subme 6 --partitions all --8x8dct --trellis 1 --threads auto --progress --sar 1:1 --no-psnr --no-ssim" --output "%encoder_out_file%" --avs "%source_dir%%target_name%.avs"

Which makes for an example....

"C:\Controller\Controller.exe" --first "--bitrate 920 --level 4.1 --bframes 3 --b-pyramid --threads auto --progress --sar 1:1 --no-psnr --no-ssim --subme 1 --partitions none --me dia" --second "--bitrate 920 --level 4.1 --ref 3 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --subme 6 --partitions all --8x8dct --trellis 1 --threads auto --progress --sar 1:1 --no-psnr --no-ssim" --output "R:\Vandread\Vandread s01-e01\Vandread s01-e01_EncoderOutput.mkv" --avs "R:\Vandread\Vandread s01-e01\Vandread s01-e01.avs"

Drive R:\ is a shared drive mapped at every computer to R

if from console the you could see x264's error.. if from satxrip - then you will not see the message...

Looking at the out-dump.txt this is the error reported.

2008-10-11~19:08:13.10 218 2nd pass frames 244 to 273
2008-10-11~19:08:13.18 218 done recieving stats
2008-10-11~19:08:13.20 Testing file "R:\Vandread\Vandread s01-e01\Vandread s01-e01.avs": FOUND!
2008-10-11~19:08:13.20 218 agent-based encoding
2008-10-11~19:08:13.21 218 doing the following:
nice -n 10 "x264.exe" --bitrate 890 --level 4.1 --ref 3 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --subme 6 --partitions all --8x8dct --trellis 1 --threads auto --progress --sar 1:1 --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --bitrate 11 --fps 25/1 --pass 3 --stats "C:\Agent\stats 94473f.txt" --seek 244 --frames 30 -o "C:\Agent\output f8b890.mkv" "R:\Vandread\Vandread s01-e01\Vandread s01-e01.avs"
2008-10-11~19:08:28.65 218 exited -1073741819
2008-10-11~19:08:28.67 Thread received exception "Failure(\"ERROR: nonzero return code from x264 (-1073741819)\")"
2008-10-11~19:08:28.75 closing socket
2008-10-11~19:08:28.75 socket closed

so.. or you kick staxrip-writer to implement "suppressors" (it's just 1 line of code on delphi) .. or you wait for new x264farm (i've told omin this issue.. long time ago).. or i'll look for my "nice.exe" somewhere here ...
"suppressors" must be implemented in a higher level application..
like staxrip-agent -then-> x264farm-agent -then-> nice.exe -then-> x264.exe

delphi is WAY out of my leage, as well as most programing (haven't done any since C++ 10 years ago hehehe) so there is nothing I can do in at area hehehehe

SpAwN_gUy

13th October 2008, 09:31

Agent is running on every computer (3) from a console (started from a shortcut).

Looking at the out-dump.txt this is the error reported. out-dump.txt will not give you the resuls i mean..
you should look on the console-window, where agent.exe runs.
'cause x264.exe outputs information direct to console. and out-dump.txt does not have this information. while "console window" still has the information.

Nakaleen

2nd December 2008, 22:16

out-dump.txt will not give you the resuls i mean..
you should look on the console-window, where agent.exe runs.
'cause x264.exe outputs information direct to console. and out-dump.txt does not have this information. while "console window" still has the information.

Desided to give this program a try again....

And again it is giving me a head ache from all the keyboard slamming.

3 computers, 3 diffrent setups, a mixture of XP, vista, Intel, amd, laptops etc... no matter what setup they all crashes at some stage. This is very annoying as I set up to encode 5 videos overnight and wake up to see it crashed at the end of the first one.

I copied this from the last render off one of the agents, that caused a crash.

815 2nd pass frames 35531 to 35531
815 done recieving stats Testing file "R:\Read Or Die\Read Or Die S01E02\Read Or Die S01E02.avs": FOUND!
815 agent-based encoding
815 doing the following:
nice -n 10 "x264" --bitrate 917 --level 4.1 --ref 3 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --subme 6 --partitions all --8x8dct --trellis 1 --threads auto --progress --sar 1:1 --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --bitrate 1853 --fps 25/1 --pass 3 --stats "C:\Agent\stats 24b0cf.txt" --seek 35531 --frames 1 -o "C:\Agent\output b6babc.mkv" "R:\Read Or Die\Read Or Die S01E02\Read Or Die S01E02.avs"
avis [info]: 640x480 @ 25.00 fps (35532 frames)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 Cache64
encoded frames: 1/1 (100.0%), 4.02 fps, eta 0:00:00
815 exited -1073741819
Thread received exception "Failure(\"ERROR: nonzero return code from x264 (-1073741819)\")"
closing socket
socket closed
Thread 822 killed on uncaught exception Unix.Unix_error(40, "send", "")

It is not much diffrent then what is in the log files. So I am stumped that is causing the error. Searching for error code -1073741819 for x264 found nothing. Maybe it is one of my x264 setting... but I can't find any info on what setting could crash all my systems.

Ichling

2nd December 2008, 23:00

Hi Nakaleen,
can you give me the exact options that you used to start the controller with? I was running in similar problems with my first tries at x264farm(-sp) and they, in the end, where all related to the x264 settings i've used.

Jabroni

1st February 2009, 18:14

So has anyone done any custom build of x264 thats a bit newer to current SVN release ? I want to make DXVA compatible encodes, and I think I need some parameters that the old x264s doesnt have (or at least the one that works with x264farm).

SpaWnGuY: Are you still working on your megui-mod? I accidently 'upgraded' the megui install, and now it removed all the x264farm stuff, and I cant find the installer of it =\ Would you mind posting it again please?

:thanks:

Pulp Catalyst

12th February 2009, 15:28

hopefully when BD Re-Builder is released someone will design a Farm program like this one
http://www.afterdawn.com/guides/archive/dvd_rebuilder_tutorial_advanced_page_8.cfm

RB-Farm was brilliant and worked great, why you ask, because the way DVD Re-builder pulls apart the DVD,
hopefully BD Re-builder will do the same,

Jabroni

12th April 2009, 19:38

Was wondering if there is going to be any update to this project and/or MeGUI. Would love to distribute all my new encodes with x264farm (im using parameters used on newer x264.exe build, and for some reason they make the x264farm clients crash)

:thanks:

Allbestmessage

15th April 2009, 10:30

There is a command line encoder profile you could use.

Breakthrough

24th July 2009, 18:47

Hello, I have a few questions...

What build of x264 do you all reccommend to use with x264farm? Or at least, what was the final build before the revision to the .stats file was committed?

Also, does the server computer also have the ability to render? Could I theoretically run a controller AND an agent on the same PC?

Thanks!

nst6563

28th July 2009, 19:50

I'm itching to get this working as designed...however it seems I've either got something setup wrong, or something is setup wrong :p

To start, I've got the controller running on a 3Ghz Core2Quad with 6GB ram. I have another machine that is connected via 100mb switch which is a core2duo w/4gb ram.

I've got the config setup for adhoc, and the controller and client see each other - that's not the problem.

The problem I think I've run into is the speed. I was hoping it would be faster than standard encoding with something like ripbot or handbrake...however I get about half the speed compared to those gui-based encoders.

I've tried the controller-based encoding as well as the client-based encoding. I don't remember if I got the client based encoding to work correctly tho... the way I read and figured out was that each client needs access to the sources being encoded right? I made a mapped drive on the machines, and pointed the config to that and it seemed to work ok.

I have 5 machines with a total of 13 cores that I could put towards something like this...I'd love to see it scream though an encode...what could be my limiting factor other than the 100mb switch (trying to get a 1000 but my current 1000 switch shot craps...thanks Dlink)

any plans of something like a gui for the controller setup? Drag a bunch of files to encode to the window (or provide a path with the files to encode), answer some questions and let it go. maybe launch the agents as a windows service?

iplmedia

15th January 2010, 06:17

I'm looking to bring this code up to date with the latest x264 and to bring it to maturity. If anyone, Ideally the author, is able to help in this please let me know.

creamyhorror

15th January 2010, 16:10

I'm looking to bring this code up to date with the latest x264 and to bring it to maturity. If anyone, Ideally the author, is able to help in this please let me know.
Don't forget the single-pass variant: x264farm-sp - single-pass distributed encoding (http://forum.doom9.org/showthread.php?t=136053), which was last modified in 2008 I think.

creamyhorror

18th January 2010, 15:54

I'm looking to bring this code up to date with the latest x264 and to bring it to maturity. If anyone, Ideally the author, is able to help in this please let me know.
Another guy is interested in developing distributed encoding for x264: Distributed Encoding with x264 (http://forum.doom9.org/showthread.php?p=1364155#post1364155)

bnshrdr

18th January 2010, 20:18

Another guy is interested in developing distributed encoding for x264: Distributed Encoding with x264 (http://forum.doom9.org/showthread.php?p=1364155#post1364155)

I really do like the idea and am fully capable of the actual design of the program. The only thing I truly lack is knowledge of x264. Sure I use it for all of my encodes, but I don't know enough about it to write pretty much a wrapper program to utilize it.

Currently I have 2 computers here at my home encoding the same video file, but some of the ways I am achieving this are either ugly or impractical. Also I don't really know how to truly handle a multi-pass scenario.

I just need info on how to do faster seeking, like using an index. I know of DGAVCDec but I don't know how to put it to good use.

I also need a way to get the frame count of the source before processing the job, but my knowledge of the AVISynth API is 0.

I am also welcoming any ideas about the whole design of this program.

iplmedia

19th January 2010, 21:17

I really do like the idea and am fully capable of the actual design of the program. The only thing I truly lack is knowledge of x264.

If am very familiar with x264 just not how to wrap it! x264farm has many benefits to it by dividing the video into GOP, however, it decompresses it to ray YUV before compression (From what I understand) and this is done in the first pass that only 1 machine performs. Thus greatly slowing down the encoding process. I would love to discuss the possibility of working on this project if you like. Send me an email and we can discuss this.

creamyhorror

19th January 2010, 23:58

Great, hope you guys get together to start a project. You might want to hang out in #x264dev on Freenode IRC to ask the x264 devs questions.

iplmedia

20th January 2010, 20:47

I have worked with and am working with one of the x264 developers already on a different project. They are a great team of developers!

bnshrdr

21st January 2010, 04:36

As much as I'd like to start a little project, I really don't have time. During the school year I always nibble little bits away from my project goals. I'm kind of just a hobby programmer, but being a full time student limits the time I can contribute to any sort of project. Thanks though for the offer and I appreciate your support. I know where I'll come when I have problems :P.

popper

21st January 2010, 13:39

the Very first thing you should perhaps do is totally do away with a fixed x264 version requirement in any code so as not to run up against the code updating being dropped as happened with farm for SO long...

it might be a very good idea to start by parsing a current x264 --fullhelp at runtime and populating your input/output variables from that parse, so that as fullhelp grows over weeks,months or years your code can take advantage of any new options that come along later and still work fine when things change.

as it is today the general feeling is that always using the constant quality option CRF= as your base is the default way to go.

x264farm while a good idea always bugged me in the way it doesnt seperate out all the different parts of the operations code into self contained stand alone scripts, so making it really hard to add new or just different ways of doing the same thing for a given situation,as a plug-in type way to work, so that could be made far cleaner if YOUR really serious about this rework.

you might want to add a new better client/server part but keep all the other bits as they were for instance, or you might want to add DHT for the auto what Hw's available to our IP cluster at this time, and who's ready to take another job, send a DHT message and add and/or act to the 'message based' actions que(s) etc... the options are endless with a little thought and an open self contained fully commented script for one action, or were needed for simpicity a set of related fine grained actions etc.

it should also be considered that current x264 also takes many input file types stand alone now OC so thats a good thing, and sooner or later it will also allow audio parsing/re-encoding to a given container too, so making the whole updated x264farm framework far easier and better later IF YOU JUST PARSE a current x264 fullhelp output or ask DS to provide a better way perhaps if your serious about distributed encoding.

for instance how might a future x264 farm make easiest use of things like this guys http://forum.doom9.org/showthread.php?t=152232 with HPC cluster (750 cores)...

people will always want to use the latest x264 so you had better find an easy way to always auto include/use it with a new x264farm framework without braking the farm....if you want people to use it and give lots of thanks and perhaps even get involved for years to come ;0)

and preferably cross platform usable too, put it infact the whole turnkey package on a generic slax liveCD http://www.slax.org (WITH DEFAULT ready setup pxe server LAN booting AS STANDARD BTW) AND boot any hardware thats in the LAN ANYTIME.

just a few thoughts to consider anyway.

Karyudo

22nd January 2010, 16:53

Good timing! I just started researching how to build some kind of x264 render farm, and here I see new interest after what looks to be a year or more of downtime and benign neglect. Nice!

I can offer almost zip in the technical department, but I do have a home network with four (soon to be five?) computers that could conceivably all run DGDecNV on at least Core2Duo processors, so maybe I'd be able to help with beta testing (when things progress that far, I mean).

Or, if that's no big help, then at least let me add my vote for this idea as a project that I think would be powerful and useful!

iplmedia

22nd January 2010, 21:05

Thank you for your thoughts on this project. I am currently planning out the best architecture for this project trying to utilize the best possible means to accomplish the best possible product. I agree with future compatibility as well as cross-platform. I currently work solely on a mac and would like to see it work on that but understand their are many other users in the *nix and windows world. The concept of a liveCD for the cluster is very familiar to the Linux world (Clusterknoppix etc).

I am considering using OpenMPI and dividing the video into GOP's for distribution so regardless of how many videos you have in queue the entire cluster will be utilized. I realize that their are inherent flaws and considerations but my goal would be very fast cluster detection, minimal bottle-necking and intelligent distribution amongst all available resources with fault-detection and load balancing.

It's a tall order but in my opinion much needed and sought after.

Please continue to post your thoughts/ideas/suggestions.

Thanks

kemuri-_9

23rd January 2010, 02:37

nitpick: OpenMPI is only an implementation of MPI-2, you would be using the MPI-2 API if using OpenMPI as a test implementation.

I had worked somewhat on an MPI-2 patch (was using the MPICH2 implementation myself for windows) nearly a year ago for x264 but stopped eventually when i realized that it would require a fairly large restructure.
I had also planned on GOP distribution among the cluster and was coding along that guideline.
it would probably be easier to implement now utilizing threaded lookahead that was committed some moths ago.

magnatique

24th January 2010, 18:05

when this is up to a testing phase, I would be happy to give the farm some testing grounds as we have core2, quads i7s at the office on our gigabit lan, with some on XP / win7 so it should give a good testing ground!

Phil

iplmedia

24th January 2010, 22:14

Hello all,

Thank you for your input. I am continuing to research and gather the best possible method to accomplish this and your input helps eliminate dead-ends and un-needed wastes of time etc. So please keep them coming. In regards to the MPI-2 implementation I still see it as the best method as well as the GOP. What were some road-blocks you ran into with using MPI and would you suggest a different method or is it still be best method just that it requires a lot of work. I am willing to invest the work/time/investment that it would take to accomplish this I just want to make sure I'm heading in the right direction. Would you be willing to send me the work that you have started already on the x264 MPI implementation? Any thoughts and information you can pass my way would be greatly beneficial. Cheers

Aaron

kemuri-_9

25th January 2010, 00:56

hmm.. having to pick my brain as to the thought processes i had back when i was working on it...

the MPI-2 API has some functionality to determine rank in the operational cluster (along with the size of the cluster) and i used this to define the following:
a node with a rank of zero would indicate the master node - that is the node that is controlling source decoding to send and receive frame data back from slave nodes for writing the final encoded video
otherwise a node has a rank of nonzero and would be considered a slave node.

A) Where to decide where to transfer frames between master and slave nodes
When i was working on this i had issues on where I was going to do this and was a major factor in stopping work.

A1) from working on the threaded lookahead patch,
I would think it would be easiest location for transfer frames from the master node to the slave node.

A2) as for the reverse.... don't recall thinking about that much... thoughts now would have it either in encoder_frame_end or in x264cli

B) what requires transferring:
originally when i was working on this there was only 1 frame type, but now with there being 2, this changes up a little bit.

B1) fdec x264_frame_t
frames that have been slicetype decided, would be transferred from master to slave for encoding.

B2) fenc x264_frame_t?
frames that have been encoded and QP decided, would be transferred from slaves to master.
this is a bit tricky as x264_frame_ts have a lot of data that would be unncessary at the master slave as all it would really want is the resulting x264_picture_t.
though if you only do decide to send back the x264_picture_t you would have to special case the --dump-yuv option in some fashion.

B3) stats
information acquired while processing frames,
involving bit costs and other information used at encoder_close and in stat file writing,
would need to be synced between master and slave node (i didn't get to this part so not sure what the best way of doing it would be).

B4) x264_param_t
the parameters to init the encoding engine with,
I had ambiguous thought patterns on how to handle this,
but i think my last choice was to init the master node with the param_t given on the cli and then send a possibly altered param_t to slaves to init with.

C) Gotchas i remember
C1) data sending.
transferring structures is eloquent in MPI-2 especially when they have pointed to data.
this required creating an extra send buffer and `packing` along data in a struct referred by pointers along with the struct to be unpacked at the receiving side.

C2) GOP distributed encoding.
using a GOP distributed encoding style would allow slave nodes to process on different GOPs with requiring relatively no information from each other.
however, this does have some down sides:
-- a) this could potentially cause the master node to allocate a large amount of data to buffer depending on how transferring of data is done.
that is if choosing to transfer all the GOP in bulk then this would require a buffer large enough to send and recv it all. naturally this could be alleviated by transferring in smaller pieces (e.g. mini GOPs).
-- b) requires unique GOPs to exist. that is the GOP distributed encoding fails when there are not unique IDR initiated GOPs, like in the case of Periodic Intra Refresh (PIR) or Open GOP.

C3) different CPUs and related issues
as x264 works on several different CPUs you would think "oh i could have this cluster over all the different cpus it works on easily", which is sadly untrue.
x264 uses the CPU's cacheline to determine how to pad frame data in x264_frame_t's for optimal performance on that cpu.
there's a few ways you could handle this when clustering between unique cpus:

-- a) alter all the node CPUs to have the same cacheline values as the cpu within the cluster that has the largest cacheline performance value
(this would require using more memory for the cpus that had the cacheline padding increased)

-- b) alter all the node CPUs to have the smallest cacheline values of the cpu within the cluster
(this would require using less memory for the cpus that had a cacheline padding decrease at the cost of some performance)

-- c) repad the data to the receiving cpu's cacheline at send/recv ing of data
(but seriously this would be more effort than it's worth so i don't recommend it)

tl;dr:
lots of things to worry about and the finesse required to do it was a major factor in me stopping work on it.
(didn't help i was in my last semester of college while working on this and wanted to graduate more than work on it)

overall i didn't see a particular issue in using MPI-2, as the surrounding issues are not transfer protocol specific but sync specific.
MPI-2 does offer a variety of sync/async API to help out on the issue of synchronizing, so i would continue recommend using it as the implementation to do this cluster support work.

I would be willing to share the code i have, but as it is incredibly old and imo bleh now,
I would only use it now as reference code on how to do something rather than actually trying to apply it to current repository and working from there.

popper

25th January 2010, 01:02

for more input you might also look to spending some idle time on http://gogloom.com/client2/index2?mainCHAT=1&network=FreeNode&channel=%23x264 to.

i still think using a generic slax is a good way to go , incase i didnt make it clear you download a generic slax livecd, boot it, select the setup pxe server LAN booting option, and instantly you have a lan bootable slax, go over to any other PC on your LAN AND boot it into pxe mode and it will be running that exact same livecd you just booted with every module that you happen to have put on the slax livecd....

slax is the only generic liveCD/liveUSB distro i know of that comes with this pre-setup pxe setup and ready to use out the box, anywere.

OC its werth also mentioning that slax creator Tomas M is also the guy that gave you the liveCD/liveUSB scripts that everyone uses today OC

http://www.slax.org/forum.php

you could boot a native liveCD/liveUSB or just use a Vbox and liveCD/liveUSB that for background client/server LAN and Vlan use etc with all the farm and related apps in module form you require on there for on the fly mouting/loading , no messing about.

popper

25th January 2010, 03:34

"I currently work solely on a mac and would like to see it work on that but understand there are many other users in the *nix and windows world."

thats not such a big problem though,(PPC G4/5 OR X86 BTW?) all it takes is picking the right tools for your potential client/server tunnel apps to the cluster you might use....

using 'rebol view' and 'rebol core' for the GUI scripting and client/server tcp:ip udp and multicast cross platform apps today. http://re-bol.com/rebol.html http://www.rebol.com/download.html is one option to consider if your into cross platform scripting control and related apps for instance.

heres a self contained multicast rebol view script that shows you how to make and use a simple multicast whiteboard for realtime data drawing and passing ,and more.
http://www.rebolfrance.info/org/articles/multicast/sources.zip
google translate his french page if you like
http://www.rebolfrance.info/org/articles/multicast/multicast.html

you want a tiny self contained pro webserver at the heart of this cluster(s) then include The Cheyenne Web Server
http://www.rebol.com/news/cheyenne.html written in rebol, or want a cache proxy for the potential message parsing etc then
use this as your basic outline and change it as required.
http://www.rebol.org/view-script.r?script=proxy.r

perhaps you want to use java as well for part of the new farm, then theres a DHT and building a multicast protocol on top of Bamboo paper by Marcel Dischinger outlining how you might include that http://bamboo-dht.org/tutorial.html

want a basic self contained multicast ipv4 overlay tunnel for connecting remote farm sections then consider the java 'mtunnel' , alas he took the page down now but its still available...

as i said the cross platform options are many if you just consider your requirements and can figure out the best and various ways to chop up the core HD content.

how about you draw up a basic visual flow chart jpg outlining your current thinking as to all the different sections that need laying out and posting that as a start for people to see and comment on any obvious potential problems ?, nothing fancy, just a quick visual aid.

iplmedia

25th January 2010, 03:49

Thank you all for your excellent input and interest in this thread. I will consider compiling a more detail brain mapping of my thoughts for critique here. Of course I need to find the free time. At any rate, Great suggestions. Keep them coming.

Aaron

popper

25th January 2010, 04:54

OC allowing for the inclusion of a cheap freeNAS http://freenas.org/freenas or 3 :) and simple Ethernet iSCSI setup etc is always a good thing to take into account for these types of mass data storage LAN Processing Farm jobs too as its quicker than other protocols over ethernet.

http://www.microsoft.com/downloads/details.aspx?familyid=12CB3C1A-15D6-4585-B385-BEFD1319F825&displaylang=en

http://freenas.org/documentation:setup_and_user_guide:iscsi_target

http://frrl.wordpress.com/2009/08/25/fun-with-freenas-iscsi-when-a-local-disk-is-not-local/
http://www.windowsnetworking.com/articles_tutorials/Options-creating-software-based-Windows-iSCSI-Server.html

SpAwN_gUy

26th January 2010, 09:42

hi guys :) .... i'm not totally back, but anyway :)

i just thought about "how's it going on my fav x264farm?" and, viola, the thing are moving :) (i thought about "fixing" farm to be able to work with newer x264 builds. but that was just a thought :( )
i'm (as a x264farm support-unit) currently switched my job(i'm a php-dev now), so now i don't have access to a variety of HW, but i do have some ideas (if someone)... and i'll keep monitoring.. 'cause i do have alot of AVC-HD-fromMyCam-Footage to encode.. and his is a real pain in the cpu.

greets from Belarus.

popper

23rd February 2010, 12:53

well that interest didnt last very long it seems, given the TOTAL LACK of any code updates or even progress posts so far...

the intent to make something better is only as good as the coding work you actually put in and release.

Yiu

4th November 2011, 04:59

Hey Spawn Guy.
I'm thinking of getting a farm server with a number of AMD Opterons running Ubuntu/WinServer2008 and controlling it through my Win7 (x64) desktop. The problem is, I have no idea where to start when it comes to setting up the server, x264farm, and all that.
I've been using x264 for a few years now, and I've perfected a rather nifty configuration but I only get 0.45FPS on my Intel i5 750 @ 3GHz setup so I pretty much NEED the server otherwise I spend days on a single conversion.
I know I should get the system built, have a Linux/WinS version of x264 on the machine, then figure out how to set up your x264farm/MeGUI mod to work with it but I'm clueless and to be honest, scared of spending thousands on a server which may go to waste if I don't figure things out. :L

Can you give any pointers as to where to start?

SpAwN_gUy

4th November 2011, 10:17

Yiu

4th November 2011, 10:25

the good thing is, that it's not so hard to set things up on a winPC.
the bad thing is... x264farm is INCOMPATIBLE with current x264 builds :(
i think the latest version supported is.. ~r900 or so..
so until someone update x264farm code to support parsing of current .stats file... i guess you'd have to stick to 1PC encoding :(

Thanks for getting back so quickly. I don't know how stupid this sounds right now but would it be possible to run x64 x264 in Win Server?
I'm probably getting the i7 3960X and OCing to ~5GHz but that's still not good enough for the amount of time it'll take.

If not,does the *nix variant of x264 support the same command-line arguments? I'm pretty sure 48 logical cores running at 2GHz will be worth getting the workhorse anyway. If possible, I'd like to raise the clock speed on the Opterons too.

48 cores X 2 GHz = 96 operations /s
V.S.
6 cores X 5 GHz X 2 (HyperThreading) = 60 operations /s

Then again, the i5 2500 beats the AMD 8 Core bulldozers so it's iffy anyway...

Sorry to hear about the compatibility problems. :(
I had been looking forward to setting up x264farm for a few months but I'm not sure if my settings file will work with the versions of x264 your application supports. If it does then it shouldn't matter, but if it's not then I'll have to build a multi-rack system instead of networking lol.

4th November 2011, 11:39

Thanks for getting back so quickly. I don't know how stupid this sounds right now but would it be possible to run x64 x264 in Win Server?

Yes.

I'm probably getting the i7 3960X and OCing to ~5GHz but that's still not good enough for the amount of time it'll take.

What kind of encoding settings and filters are you tormenting yourself with?

does the *nix variant of x264 support the same command-line arguments?

Yes. But there's no native AviSynth support on other operating systems than Windows. You need Wine and avs2yuv for that.

48 cores X 2 GHz = 96 operations /s
V.S.
6 cores X 5 GHz X 2 (HyperThreading) = 60 operations /s

HyperThreading gain is x1.5 at best.

Sorry to hear about the compatibility problems. :(
I had been looking forward to setting up x264farm for a few months but I'm not sure if my settings file will work with the versions of x264 your application supports.

I'd advice against using such an old x264 version even if it supported all your settings.

Do you need to finish one encode as fast as possible, or could you simply encode separate works in parallel?

Yiu

4th November 2011, 12:01

HyperThreading gain is x1.5 at best.
Yet the 4 core i5 2500k thwarts the AMD FX 8 core processor, let alone the 2600k, 2700k, or the awaited 6 core 3960X.

What kind of encoding settings and filters are you tormenting yourself with?
A near lossless setting. No filters. Straight BluRay/H264 to x264 compressed video files. Retains approximately 99.8 per cent visual quality for animation and approximately 95 per cent visual quality for live action. It's fine-tuned for anime essentially.
Can take up to 8 days to compress a legal back-up of one of my average BluRays. I'm running out of space, so spending days getting near lossless quality AND rather small file sizes is worth it.

Do you need to finish one encode as fast as possible, or could you simply encode separate works in parallel?
Well when one whole film, lets use Transformers 2 as an example, can take up 16 days and still comes out 12GB (with DTS 5.1 in 3 languages and many other audio streams), I'd love to split it up on several machines. It's not about NEEDING speed (I.E.: Realtime), it's about not waiting for days with 100% CPU usage. If I could get encoding time for film down to 1 day (12 hours per pass, excluding audio), I'd be happy.

With this same settings file, I edited a little clip here (http://www.mediafire.com/?monpic74ip7hn9a) which is completely original (well, aside from the actual DTS Logo). It's 3.27MB and is in 1920x1080 @ 23.976FPS with FLAC 16 bit 48KHz 5.1 surround audio.

The video data for that file is only 516KB. Granted there's only 1 Keyframe and limited chroma motion, but the point is for cel and digi-cel animation it's a distribution foothold. I, by the way, am working on an animated series which is what I plan to use this for.

It's starting to look like there's no good solution.
Isn't Win Server limited to 2 physical CPUs at the moment? Or is that the previous edition?

vivan

4th November 2011, 12:37

Yet the 4 core i5 2500k thwarts the AMD FX 8 core processorAMD FX also doesn't have real 8 cores. So it's not HT win, it's AMD fail ;)

Isn't Win Server limited to 2 physical CPUs at the moment? Or is that the previous edition?http://en.wikipedia.org/wiki/Windows_Server_2008_R2#Editions - it says that it support from 1 to 64 physical CPUs depending on edition.

Atak_Snajpera

4th November 2011, 12:44

AMD FX also doesn't have real 8 cores. So it's not HT win, it's AMD fail
AMD FX has 8 integer units and only 4 FPU. In fact it looks more like 4 core cpu with doubled number of integer units.

4th November 2011, 13:22

A near lossless setting. No filters. Straight BluRay/H264 to x264 compressed video files. Retains approximately 99.8 per cent visual quality for animation and approximately 95 per cent visual quality for live action. It's fine-tuned for anime essentially.
Can take up to 8 days to compress a legal back-up of one of my average BluRays. I'm running out of space, so spending days getting near lossless quality AND rather small file sizes is worth it.

0.5 fps on an i5 750 sounds like a lot of time wasted for very minimal gain. You are using placebo-level settings which probably give you 1-5 % more compression than --preset slower would, while taking 5 times longer to encode.

Or put in a different way, you can get the same quality by using:

A. --preset placebo settings that give you a 5000 MB file in 48 hours

B. --preset slow/slower settings that give you a 5250 MB file in 10 hours.

Is that 250 MB really worth it? At current HDD prices it costs less than 2 cents!

I seriously suggest spending the money on HDD space instead of CPU power. With the amount of money you'd spend on server hardware, you can get enough HDD space that you can make 1:1 copies of all the Blu-rays you'll ever own!

Well when one whole film, lets use Transformers 2 as an example, can take up 16 days

Ok, you're not using placebo but insane settings?! What did you put in that command line?

If I could get encoding time for film down to 1 day (12 hours per pass, excluding audio)

Use CRF instead of 2-pass. Or if you absolutely need to control the file size (why?), use fast first pass.

Isn't Win Server limited to 2 physical CPUs at the moment? Or is that the previous edition?

2008 R2 standard edition supports 4 physical CPUs: http://en.wikipedia.org/wiki/Windows_Server_2008_R2#Editions

Yiu

6th November 2011, 03:40

Is that 250 MB really worth it? At current HDD prices it costs less than 2 cents!Yeah it's really worth it. Esspecially when I'm already running low with 6TB of data already. I need another 2TB Caviar Black but the prices over in the UK just spiked. Prices went up nearly 50% on HDDs, but SSDs haven't gone up. About $225 USD for 1 WD CB 2TB.

I have a lot of data, and it's going to be expensive when I decide to raid all my drives.

Ok, you're not using placebo but insane settings?! What did you put in that command line?I spent months looking into the x264 settings. Anything to reduce size without reducing quality I put in. If it ever compromised quality, I'd bump something else up that bumped up analysis and vector encoding.

Use CRF instead of 2-pass. Or if you absolutely need to control the file size (why?), use fast first pass.
Well I might as well be setting CQ and setting it to 2-4. But they're really selective settings which I've looked at and used, CQ doesn't give as good visual benefits, in my opinion.

2008 R2 standard edition supports 4 physical CPUs.
Yeah as I thought, my information is kinda behind. Since x64 x264 can run on WinServer on 4 CPUs on just standard I'll have no problem starting out. By the time I can afford more racks I'll be able to afford Ultimate.

EDIT:
I'll probably need HPC/Advanced for the RAM headroom. Unless x264 doesn't actually use that much memory? I'd assume it'd use more to speed up encoding time but I don't know.

Atak_Snajpera

6th November 2011, 10:26

Yiu

6th November 2011, 14:26

did you calculate how much money you waste for electricity? under havy load your pc probably eats about 250-300 W. i would rather buy extra hdd. if you compare average salary from uk and from eastern europe you will quickly realize that prices are not that bad. we from east earn four times less than you from uk. please stop wining.

I'm from Michigan USA. I don't qualify for a lot of free shit other people do over here. Look at average salary all you want, I'm an 19 year old student. And in the end, I'm using less HDD space, buying less HDDs and thus saving the amount of power I use in the long run. (Keep reading it gets better.)
That's why I'm looking to buy a workhorse, so that the 1000+ watts I'm using* can be used to finish the video compression, rather than on my regular desktop system which has disk drives, graphics cards, external sound cards, and a whole lot more than the workhorse will ever have. More energy can go straight to RAM and more processing power, I actually save a lot of money getting a video compressed with a workhorse.

*The main point is, I can only have 1 of the computers on at a time to actually break even. But, if the workhorse can do the work faster than the desktop and either computer would be more or less useless while the CPUs maxed out, then I'm saving time. Swapping the money I save for more time.

It's extremely useful for distributing my own content as well. In the UK your upload speed is limited to 1/8th your download speed on most connections. I only get a 1MB/s download rate (8Mbits) so a 128KB/s upload rate can restrict deadlines and file transfers for me. In all honesty if I had a monster internet connection and more money to throw at HDDs, I'd be far less worried about it but with the rest of the world still on slower internet connections it saves time not only for me but thousands of other people.

Atak_Snajpera

6th November 2011, 14:56

Still simple remuxing from HD-DVD/BD/AVCHD is better because:

1) Quality is 100% the same with source
2) Speed is only limited by your HDD/SSD
3) lower electricity bills

and now what you get:
1) Smaller size
2) Quality will never be at 100% with source
3) Ridiculous encoding speed even at good system (0.45 FPS!?!!)
4) A lot higher electricity bills due to have load on PC.

I see no logic here.

Yiu

6th November 2011, 15:58

You have noooo idea how much media I have. With minimal quality loss and most BluRays getting shrunk to 10GB including specials and extras, it's worth it.

I would need about 6TB of space just for videos if I did direct copying. Then I've got to factor in my other 4TB of ever expanding media types. As I said, I'm gagging on 6TB of total space already. If I could, I'd get a media server but that would eat up tons of electricity, and all for what? The ability to access all my data at the once? Plus, how much more data will I eat up just over the next 12 months?
I only started digitizing my BluRay and CD and other mediums in February. I've not even been doing this for 12 months and I'm at 6TB.
I was looking to not buy anymore HDDs until next year last month when I bought my Caviar Black but things aren't going as well as I planned in terms of space.

I'm already buying the Desktop, Workhorse and a camera for college all at the same time with my school loan and grant. I have no money to buy any HDDs. I'll need to find a job or get my business up to a decent pay-grade to be able to keep up with my current HDD purchase intervals.

jasonwc

4th December 2011, 17:16

Your arguments really don't make any sense. I'm going to prove it by breaking down the cost of your alternatives. First let's start with your hardware choices.

You purchased the $1,000 Core-i7-3960K processor and then overclocked it to 5 Ghz despite the fact that the Core-i7-3930K could have been overclocked to the same rate and would have performed within 1-2% for a savings of $400. And I'm betting you spent at least $300 on the motherboard. So that's $1,300 on two components.

Compare that to a Core i7 2600k ($280 at Microcenter) and a decent 1155 board ($150) for $430. You could then take the extra $870 and put it into drive space. Even at the outrageous prices you can buy FIVE 2 TB drives providing 10 TB of storage for that price (Best Buy has a 2 TB WD20EARX for $170).

Or, if you had reacted to the impending price hike, you could have bought a bunch of drives at $80 or less in October like I did. . .

I have a 4U Norco RPC-4224 file server with 24 hotswap bays (currently using 9 bays - 16 TB useable, 2 TB parity) and I only spent $700 to put together the server. I buy drives as necessary. I acquired two Hitachi 5K3000 2 TB drives for $60/each before the hard drive price surge, and then two Western Digital WD20EARX drives from Best Buy for $80 afterwards. That's an additional 10 TB (9.1 TiB) of storage for $280. Even at around 35-40 GB/movie and accounting for TB --> TiB conversion, you could store 250+ movies on those drives.

But if you really insist on encoding rather than remuxing, you should be getting 5-6 fps in second pass or more with your 5 Ghz hexa-core setup.

You're spending 5-10x more encoding time than if you had used placebo settings for a ~1% efficiency improvement, and in so doing are wasting MANY Kwh of electricity. You're likely burning 300 watts/hr. At 10c/ KwH or more, you would be much better off simply using a lower quality setting and slightly increasing bitrate.

For example, you say that your encodes take 8 days. With a 5 Ghz OC on a 130 TDP processor and a demanding new platform, you have to be using at least 300 watts/hr (remember your PSU is only 80-90% efficient at best).

So that's .3kW x 8 x 24 hours/day = 57.6 KwH

Assume an average price of 10 cents/kwH and you're paying $5.76 to encode a movie. You also are spending money to store that movie. You said each encode was 10 GB. Assume a price of $200/2TB hard drive. That's 10 cents/GB.

So, 10 GB x $0.10/GB = $1.00 for HDD space
$1.00 (HDD space) + $5.76 for electricity = $6.76 total

Now, consider a "placebo" encode which on your system should be at least 5x faster (1.6 days).

.3 kW x 1.6 x 24 = 11.52 KwH ($1.15)

So, let's say by altering your settings from insane to placebo you had to store an extra 1 GB per movie (probably more like 100-250 MB). That's only an extra $0.10.

$1.10 for storage (11 GB) + $1.15 for electricity = $2.25 total

In contrast, if you remuxed you would use perhaps 40 GB MAX. Even at a price of $200 for a 2 TB drive, you're only spending 10 cents/ GB. So, the remux would only cost $4.00 to store (and I'm greatly exaggerating the remux size to be conservative. Most of my remuxes are 20-30 GB - so $2-3).

So, you're looking at $2-4 for a remux depending on whether you include extras and the size of the movie, even with absurd hard drive prices.

Since I bought a bunch of drives before the prices spiked, I'm only paying 3-4 cents/GB so my remuxes cost $0.60-$1.20 to store (20-30 GB MKVs, no extras).

So, here's the cost we get per movie from highest to lowest:

1) Encoding with Insane settings: $6.76/movie
2) Remux with inflated drive prices: $4.00/movie MAX; more realistically $3.00 or less
3) Encoding with Placebo settings: $2.25/movie

BUT those prices assume your current hardware as a sunk cost. Had you purchased a Core i7 2600K system instead you would have had an additional $870 to play with. Say you took the $870 and purchased five of the 2 TB Western Digital drives I suggested. That would provide 9.1 TiB or 9,313 GiB of storage.

So, you could have had a Core i7 2600K system AND 9.1 TiB of additional storage enough for 846 slightly larger encodes or ~250 remuxes.

And with a 2600K overclocked to 5 Ghz, you would lose perhaps 30% in encoding speed. So... you could spend 2 days encoding a movie at placebo settings or 12 hours for equivalent quality with a 10% increase in bitrate.

So, if you went back and made different hardware choices, choice #3 (encoding with placebo settings) would likely cost you about $1/movie (2600k has better performance per watt) while choice #2 (remux) would effectively be FREE for the first year or two (not sure how fast you collect digital media).

The choices you have made make no economic sense. Now, if you just really enjoy having a ridiculously overpowered top-of-the-line hexacore Core i7 system, that's fine. But don't pretend that you're getting "value" from overclocking a $1,000 extreme processor or spending 8 days to encode a movie. You're not. Also, you don't need a 7200 RPM drive for bulk media storage. Get the cheapest 5400 RPM drive that's reliable.

EDIT: I misread your prior post. I thought you actually live in Michigan. It appears you live in Scotland, so I'm not sure what drive prices are there.