x264farm: distributed encoding [Archive] - Page 12

View Full Version : x264farm: distributed encoding

Pages : 1 2 3 4 5 6 7 8 9 10 11 [12] 13 14

Razorholt

11th April 2008, 15:37

Ok, I'm trying now... Always glad to help ;)

Razorholt

11th April 2008, 15:55

log start: #Apr.11.2008_9.45.a.m_Married_Life_1080p.mov#

---------------------------------------------------------------------
copying input file to tmpfiles directory

Source: C:\Videos\trailers\Married_Life_1080p.mov
Destination: C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile.mov
Buffer Size: 1,000,000 Bytes/s

---------------------------------------------------------------------
remuxing to a mkv container

command line:
-o "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile_vid.mkv" --engage allow_avc_in_vfw_mode "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile.mov"

lines returned by the mkvmerge cli:

mkvmerge v2.0.2 ('You're My Flame') built on Feb 21 2007 23:40:43
'C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile.mov': Using the Quicktime/MP4 demultiplexer.
'C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile.mov' track 1: Using the MPEG-4 part 10 (AVC) video output module.
'C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile.mov' track 2: Using the AAC output module.
The file 'C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile_vid.mkv' has been opened for writing.
The cue entries (the index) are being written...
Muxing took 5 seconds.

---------------------------------------------------------------------
creating avisynth script

script generated:

LoadPlugin("F:\[8]Coding\Ani-x264\Ani-x264\bin\Debug\avsplugins\Undot.dll")
LoadPlugin("F:\[8]Coding\Ani-x264\Ani-x264\bin\Debug\avsplugins\FluxSmooth.dll")
LoadPlugin("F:\[8]Coding\Ani-x264\Ani-x264\bin\Debug\avsplugins\Convolution3DYV12.dll")
LoadPlugin("F:\[8]Coding\Ani-x264\Ani-x264\bin\Debug\avsplugins\AutoCrop.dll")
LoadPlugin("F:\[8]Coding\Ani-x264\Ani-x264\bin\Debug\avsplugins\Decomb.dll")
LoadPlugin("F:\[8]Coding\Ani-x264\Ani-x264\bin\Debug\avsplugins\VSFilter.dll")

function logo (clip orig, string image, int start, int length, int fadein, int fadeout, int trans, int x, int y, int threshold)
{
trans = Default(trans, 0)
fadein = Default(fadein, 0)
fadeout = Default(fadeout, 0)
x = Default(x, 0)
y = Default(y, 0)
threshold = Default(threshold, 0)
framerate = orig.framerate()
overlay = ImageReader(image, 0, length-1, framerate, false)
mask= overlay.ConvertToRGB32().ColorKeyMask(trans, threshold).ShowAlpha().FadeIO(fadein,fadeout)
numframes = overlay.framecount()
origclip = orig.Trim(start, start+length-1)
newclip = Overlay(origclip,overlay,x=x, y=y, mask=mask)
return orig.Trim(0, start - 1) + newclip + orig.Trim(start + numframes, 0)
}

DirectShowSource("C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile_vid.mkv",convertfps=true,audio=false) # Source
Lanczos4Resize(704,400) # Lanczos (Sharp)

logo("F:\[6]Graphics\My Graphics\otakuleg_textlogov1 copy.png",0,300,25,25,0,5,5,0)

Telecide() # Inverse Telecine
Decimate(5) # Inverse Telecine

FadeIO2(25) # FadeIn/FadeOut Frames
ConvertToYV12()

---------------------------------------------------------------------
connecting to clients across the network

lines returned by the x264Farm TCP Controller:

Connecting to client no. 1
Connection established with client no. 1
Sending job info to client no. 1
Job info finished sending; sending agent configuration
Sending agent configuration to client no. 1
Configuration finished sending; starting x264Farm agent
x264Farm agent is running on client no. 1
Disconnected from client no. 1
Working...
---------------------------------------------------------------------
starting x264Farm controller

command line:
--first "--qp 26 --bframes 16 --b-pyramid --subme 1 --analyse none --me dia --progress" --second "--merange 12 --me umh --subme 6 --filter 1,1 --trellis 1 --ref 8 --mixed-refs --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --bframes 16 --b-pyramid --b-rdo --weightb --bime --direct auto --progress" --bitrate 280kbps --avs "InputFile.avs" --output "OutputFile.mkv" --config "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile_controller_config.xml"

lines returned by the x264Farm Controller, Agent(s) and x264 cli:

x264farm agent version 1.15-182
x264farm controller version 1.15-182
Using config file "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile_agent_config.xml"
Using config file "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile_controller_config.xml"
Temp dir: "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186"
CONFIG:
Ports: 6060-6070
temp_dir: "C:\\Documents and Settings\\encoderman\\Desktop\\Ani-x2641500\\tmpfiles\\MarriedLifemov1182793186"
x264: "x264.exe"
agent list:
nice: 10
Local = 127.0.0.1:6060 (1,1)
agents: (1,1)
bobby4 = 192.168.0.1:6060 (1,2)
agent bases: [None]
pipe: false
compressions: 0001 - 0001
deleting old files: after 604800 seconds
UDP ping port: 40700
UDP listening port: 40701
UDP name: "Local"
trying port 6060
Removing stale files from temp dir...
bound to port 6060
listening...
working!
Listener set up
WARNING: avs2yuv exited with error 1; avs2yuv responded:

x264Farm did not end as expected
Trying to recoverx264farm controller version 1.15-182
Using config file "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov1182793186\InputFile_controller_config.xml"
CONFIG:
temp_dir: "C:\\Documents and Settings\\encoderman\\Desktop\\Ani-x2641500\\tmpfiles\\MarriedLifemov1182793186"
agent list:
Local = 127.0.0.1:6060 (1,1)
bobby4 = 192.168.0.1:6060 (1,2)
WARNING: avs2yuv exited with error 1; avs2yuv responded:

---------------------------------------------------------------------
connecting to clients across the network

lines returned by the x264Farm TCP Controller:
Connection established with client no. 1
Connecting to client no. 1
Stopping x264Farm agent on client no. 1
x264Farm agent has been stopped on client no. 1
Disconnecting from client no. 1
Disconnected from client no. 1
Working...

---------------------------------------------------------------------

log end: #Apr.11.2008_9.46.a.m_Married_Life_1080p.mov#

I also noticed that I can't import AVS files. Is it correct?

kenji123

11th April 2008, 16:19

You will need to select or create a new profile. the profile it is set at isusing a custom AVS script i was using for personal usage. you can easily create or edit profiles by going to the settings tab.

It checks the script first for errors, also any files used in the script must have an absolute path (eg. not "AVISource("test.avi")" but "AVISource("D:\folder\test.avi")").

If thats not the case, can you post the AVS script?

Razorholt

11th April 2008, 17:27

I have updated my avs file. It seems to be working at that level but then, I got this:
Recent errors:
4:07 bobby4 failed with "End_of_file"
173 recv thread got exception; killing listener thread
4:09 bobby4 failed with "End_of_file"
172 first pass frames 0 to 3432
4:11 Local failed with "End_of_file"
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 SSE3 Cache64
172 controller-based encoding
4:13 Local failed with "End_of_file"
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 SSE3 Cache64
172 doing the following:
nice -n 10 "x264.exe" --qp 26 --bframes 16 --b-pyramid --subme 1 --analyse none --me dia --progress --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 2997/125 --pass 1 --stats "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\MarriedLifemov915614466\stats 6702cb.txt" -o NUL - 528x288
Last updated: 4:15
Got a connection from 192.168.0.150:1311
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
working!
Agent | FPS | Description
174 recv thread got exception; killing listener thread
Local | 0.00 | Sending [0-3432]
Thread received exception "End_of_file"
bobby4 | 0.00 | Waiting for job
closing socket
Local | 0.00 | Connected to 192.168.0.150:6060
socket closed
bobby4 | 0.00 | Disconnected
Thread received exception "End_of_file"
Recent errors:
closing socket
4:07 bobby4 failed with "End_of_file"
socket closed
4:09 bobby4 failed with "End_of_file"
4:11 Local failed with "End_of_file"
4:13 Local failed with "End_of_file"
Thread received exception ("An existing connection was forcibly closed by the remote host.\013\n",recv,)
Last updated: 4:15
closing socket
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
socket closed
Agent | FPS | Description

---------------------------------------------------------------------
connecting to clients across the network

lines returned by the x264Farm TCP Controller:
Connection established with client no. 1
Connecting to client no. 1
Stopping x264Farm agent on client no. 1
x264Farm agent has been stopped on client no. 1
Disconnecting from client no. 1
Disconnected from client no. 1
Working...

---------------------------------------------------------------------

log end: #Apr.11.2008_11.24.a.m_Married_Life_1080p.mov#

gnzrg25

11th April 2008, 18:34

Hi. Im newbie...
I'm trying to compile x264 with visual c++ 2005.
Get these errors.

1>------ Build started: Project: x264, Configuration: Release Win32 ------
1>Linking...
1>x264.obj : error LNK2019: unresolved external symbol _strncasecmp referenced in function _Parse
1>libx264.lib(encoder.obj) : error LNK2019: unresolved external symbol _x264_cqm_init referenced in function _x264_encoder_open
1>libx264.lib(encoder.obj) : error LNK2019: unresolved external symbol _x264_cqm_parse_file referenced in function _x264_encoder_open
1>bin/x264.exe : fatal error LNK1120: 3 unresolved externals
1>Build log was saved at "file://c:\Documents and Settings\Administrator\Desktop\X264_rev270_src\x264\build\win32\obj\x264_Release\BuildLog.htm"
1>x264 - 4 error(s), 0 warning(s)
========== Build: 0 succeeded, 1 failed, 1 up-to-date, 0 skipped ==========

tnx..

Razorholt

11th April 2008, 19:59

My customized AVS:
DirectShowSource("{source}",convertfps=true,audio=false) # Source
Lanczos4Resize(704,400) # Lanczos (Sharp)
ConvertToYV12()

Log:
starting x264Farm controller

command line:
--first "--qp 26 --bframes 16 --b-pyramid --subme 1 --analyse none --me dia --progress" --second "--merange 12 --me umh --subme 6 --filter 1,1 --trellis 1 --ref 8 --mixed-refs --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --bframes 16 --b-pyramid --b-rdo --weightb --bime --direct auto --progress" --bitrate 280kbps --avs "InputFile.avs" --output "OutputFile.mkv" --config "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\test2avi2147434371\InputFile_controller_config.xml"

lines returned by the x264Farm Controller, Agent(s) and x264 cli:

x264farm agent version 1.15-182
Using config file "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\test2avi2147434371\InputFile_agent_config.xml"
Temp dir: "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\test2avi2147434371"
Ports: 6060-6065
x264: "x264.exe"
nice: 10
agents: (1,1)
agent bases: [None]
pipe: false
compressions: 0001 - 0001
deleting old files: after 604800 seconds
UDP ping port: 40700
x264farm controller version 1.15-182
UDP listening port: 40701
Using config file "C:\Documents and Settings\encoderman\Desktop\Ani-x2641500\tmpfiles\test2avi2147434371\InputFile_controller_config.xml"
UDP name: "Local"
CONFIG:
trying port 6060
temp_dir: "C:\\Documents and Settings\\encoderman\\Desktop\\Ani-x2641500\\tmpfiles\\test2avi2147434371"
Removing stale files from temp dir...
agent list:
bound to port 6060
Local = 127.0.0.1:6060 (1,1)
listening...
bobby4 = 192.168.0.1:6060 (1,2)
Listener set up
working!

I see Job 1 - analyzing - [working...] at the status bar and that's it. Nothing is happening :(

RickA

11th April 2008, 21:52

To those not able to follow the thread where I was talking about the nice GUI for x264farm created by kenji123 - thanks again btw. Um, the one that he posted right above mine... Which I was referring to in my post giving the option of using the MKV container with AAC, but no option for AC3 audio in the MKV container.

Then gave a good reason why it would be nice of him to implement said option. Since just about every DVD and Hi-Def disc under the sun has AC3 audio by default. As well as the vast majority of home sound systems having the support for directly decoding AC3 Dolby Digital more so than AAC.

I know x264 is only for encoding video content. How many of you seriously watch / encode video without putting an audio in the container with it to listen to? That is why I was addressing the GUI. Have you even tried his GUI to have an idea of what I was talking about before jumping in with comments?

Cheers,
R

[P]ako

12th April 2008, 23:35

I know x264 is only for encoding video content. How many of you seriously watch / encode video without putting an audio in the container with it to listen to? That is why I was addressing the GUI. Have you even tried his GUI to have an idea of what I was talking about before jumping in with comments?

Cheers,
R

Please read your post carefully, in your addendum you mentioned x264farm and AC3, not the GUI for x264farm and AC3. And because you were speaking of x264farm I assumed you were addressing the thread poster (Omion). There is no reason to get pissy over the internet when someone seems not to understand you.

SpAwN_gUy

14th April 2008, 15:15

Hi. Im newbie...
I'm trying to compile x264 with visual c++ 2005.
Get these errors.
1st) this should go to x264 compilation thread..
2nd) I've just mastered it (again.. yes.. za sikret tikniq, 2 build x264 under Win MSVS 2005 ... even with pathces,.. and gpac and pthreads.. bwahahahaha)
ok.. i'll help..
1) convert .sln from 2003version to 2005 (as you did, as i can see)
2) add to project: "cabac-a.asm" to Core/x86/ folder AND configure it to compile with nasm as OTHER .asm's

ok. here is compilation of Mine:
upd. removed.... due to producing lotsofsh_t...

SpAwN_gUy

15th April 2008, 15:08

ok.. i've made another one... this one is a bit MORE proper :) ..

x264 rev.819, modified, MSVC2005 build
General thread:
http://forum.doom9.org/showthread.php?t=130364

x264.gaussian.cplxblur.01.diff
Dark Shikari: - gaussian cplxblur: gives a tiny improvement in 2pass ratecontrol
x264_me-prepass_DeathTheSheep.01.diff
http://forum.doom9.org/showthread.php?p=1093523
x264_2pass_vbv.7.diff
http://thread.gmane.org/gmane.comp.video.x264.devel/3093/focus=3748
x264_hrd_pulldown.04_interlace.diff
- HRD and pulldown for HD compatibility, updated patch for interlacing
http://forum.doom9.org/showthread.php?p=1047919#post1047919

grab it here (http://rapidshare.com/files/107703602/x264.819.msvc2005.modified.exe)

nvubert

22nd April 2008, 16:35

your x264farm is what i needed for some long time for my HD encodes. I'm using it about 5 months and it's really a piece of good job.
Here's my setup:
Controller:Q6600@4.0GHz (wc) 4GBDDR2-1066 2xRaptor36GB raid0 and 2xF1 320GB raid0, selfmodded XPSP2
Agents:my HTPC with E6600@3.2GHz 2GB-DDR2-800, E4300@3.2FGHz 2GB-DDR2-800 and 3xA64 X2@2.66GHz 2GBDDR2-800 ( network 1Gbps )

My question is:
Is possible to configure controller that only second pass will be processed over the agents and first pass only on my local Q6600? ( i see that v1.15 have recognize local and auto configure with agent encoding that not use av2yuv.exe )
The reason is that on 720p encodes at first pass with no avs filters speed is very slow 8-30fps, the same source on megui on local only i have 20-50fps ( depending on source )
Maybe is some trick that allow encode only 2pass on x264farm with first pass encoded on standalone x264.exe
Thanks for help

SpAwN_gUy

22nd April 2008, 16:51

The reason is that on 720p encodes at first pass with no avs filters speed is very slow 8-30fps, the same source on megui on local only i have 20-50fps ( depending on source ) you are doing controller-based or agent-based encodings?
Maybe is some trick that allow encode only 2pass on x264farm with first pass encoded on standalone x264.exe there is a trick.
you make first-pass on standalone, then start x264farm.. and after it starts doing first pass.... quit it (terminate). then locate folder (i.e. "193949954353 your.avs") in controller's temp folder, then (i guess.. i haven't done it myself) delete dir (in the dir metioned before) named like "1-34534...." and replace(or create...) file "1-BC95ECB7B64E98DEB......txt" with your .stats file form first-pass standalone encode...

but, doing agent-based is alot faster(on my side here)

nvubert

22nd April 2008, 17:11

I do controller based encoding because i can't configure every machine for share network,avs filters,etc. However since v1.15 agent on my controller self-configure for agent based encoding ( detecting avs file and says agent encoding - there's no avs2yuv.exe in taskmanager for it ) and other computers doing controller-based.

This "trick" I already knew, but i need some automate this ( if this is only solution )
It is very complicate to do this "hack" on every encode

SpAwN_gUy

23rd April 2008, 08:12

I do controller based encoding because i can't configure every machine for share network,avs filters,etc. i can... teach you how to setup agent-based... (well, there is a tutorial for mpeg2 encoding in my signature)... and i can explain everything... 'cause now i do AVC reEncodings agent-based...

nvubert

23rd April 2008, 10:40

There's no need to teach things i already know :)

Problem is that some computers can't host avs filters ( sometimes i use lots of them and scripts are heavy ) because they are HTPC's for TV 24/7 and others media in houses my neighbors, but anyway thanks for good intention :thanks:

[P]ako

27th April 2008, 19:39

Omion, would it be possible to have a new build of x264farm that includes the AQ patch (already in cvs) and the experimental film grain optimization patch?

FGO patch: http://forum.doom9.org/showthread.php?t=137117&highlight=patch

SpAwN_gUy

2nd May 2008, 13:31

ako;1129577']Omion, would it be possible to have a new build of x264farm that includes the AQ patch (already in cvs) and the experimental film grain optimization patch?

FGO patch: http://forum.doom9.org/showthread.php?t=137117&highlight=patchthere is no reason to reCompile x264farm... just replace x264.exe itself.

if you wish.. i've made r839 MSVC2005 build of x264... (but without patches)... and i'm struggling with git-apply ('cause it would be much more simplyer.. then to applay them by hands..(as i did before..))

[P]ako

2nd May 2008, 15:41

there is no reason to reCompile x264farm... just replace x264.exe itself.

Did I miss something? If I remember correctly Omion said that x264farm works only with his MSVS build of x264.

Dustin013

2nd May 2008, 20:27

I have been playing around with x264 and I am loving the codec. I have found a nice setup to encode all my Divx backups to work on my PC and IPhone which is amazing. However, I am using dual Intel 3.2ghz Dual Core Xeon processors and I can still only encode around a 3:1 ratio (about 1/3 the play time length to encode a file) when encoding AVI into a x264 format in a MP4 wrapper.

x264Farm seems like the perfect solution to speed this whole process up. However, I am converting AVIs encoded in Xvid mainly and some Divx and it would seem x264farm doesn't support AVI encoding?

Is it possible to modify x264farm to allow encoding of AVI files and possibly setup a way to batch encode?

I currently have access to some servers at work I was given permission to test on. I have 11 servers I can use to try this out. The controller server is a mammoth with 16GB ram and top of the line dual core processors. Then I have 10 dell 1U units that have 2GB ram in them each and dual 2.8ghz DC Xeon processors. If I could get the big server to work as the controller and the other 10 as agents I think I could encode my entire 3TB collection of backups into Mp4 format fairly quickly.

Any suggestions?

driftr

4th May 2008, 20:21

Dustin013: you can convert any file to x264. I've done mpeg2, avi (xvid, divx, wmv), etc etc. with x264rfarm. x264farm is just a program that utilizes the x264 codec so there's virtually no difference other than the supported build of the codec.

You just have to feed x264farm an AVS script. If you haven't used avisynth that's what you need to read up on and there are many guides at doom9 to help with that.

And one thing to keep in mind for the controller/agent computers is you'll have to install avisynth and any codecs on the computers. So you might want to be sure the powers that be are aware the process isn't a light weight application (thou it shouldn't interfere with any usual processes).

SpAwN_gUy

7th May 2008, 08:27

ako;1133563']Did I miss something? If I remember correctly Omion said that x264farm works only with his MSVS build of x264. one page back there is rev819..
i have unpatched 839... (i can post, and upgrade to current rev.)
i have linux and will try to use it for patching sources.. 'cause my conditions do not allow me to set up git properly..

[P]ako

7th May 2008, 16:05

I see. When I have time, I will try with Shikari's build of x264 with the FGO patch.

Humbula

25th May 2008, 21:36

elo,

can you tell me how you calculate the bitrate needed for a part of the second pass? is it possible to calculate is from the passfile?

Adub

26th May 2008, 02:47

Use a bitrate calculator. Just punch it into google, use MeGUI, or any other variety of programs. Again, if you google it, you'll find hundreds. And this is probably the wrong thread.

Humbula

26th May 2008, 05:22

Use a bitrate calculator. Just punch it into google, use MeGUI, or any other variety of programs. Again, if you google it, you'll find hundreds. And this is probably the wrong thread.

No, you didn't understand my posting. Let me try again: x264 farm "cuts" the movie into parts, which will be encoded by the next machine, requesting a job. Lets say one of this parts has about 250 frames.
How does x264 farm calculate the bitrate (needed) for this part?

Adub

26th May 2008, 05:54

Humbula

26th May 2008, 06:15

The bitrate shouldn't change. When you specify a bitrate in the commandline, you are saying to the encoder "aim for this". It doesn't matter if you split up the file or not, the bitrate will stay the same, as you want uniform quality throughout your encode.

x264 farm's aim in NOT to keep the bitrate for every part the same.

"The ratecontrol used is designed to be as close to the ratecontrol done by x264 itself"

I want to know how this ratecontrol works...

Adub

26th May 2008, 20:35

Oh, okay. I am sorry, I didn't fully comprehend what you are asking. And frankly, I don't know the answer. Probably the only way to find out is to look at the source or talk to Omion.

Humbula

26th May 2008, 21:32

to be honest i dont understand the code. strange language... :)

omion, help me...

foxyshadis

27th May 2008, 00:17

x264farm just processes the first pass stats files to come up with each segment's relative sizes.

Humbula

27th May 2008, 06:44

x264farm just processes the first pass stats files to come up with each segment's relative sizes.

And how can I calculate a segments size using the statsfile?

omion

27th May 2008, 06:49

to be honest i dont understand the code. strange language... :)

omion, help me...
The language is strange, but my coding style makes it even worse. You are very brave for attempting to read it :D

And how can I calculate a segments size using the statsfile?
Basically, the number of bits used by each frame is itex+ptex+mv+misc. Divide by 8 to get bytes (it should be a whole number of bytes)

[edit]If you want to know how the ratecontrol works, start by looking at x264's encoder/ratecontrol.c file. That's where I got my info. Using x264farm's source will only get you more confused. :)

Humbula

27th May 2008, 23:00

The language is strange, but my coding style makes it even worse. You are very brave for attempting to read it :D

:goodpost:

Basically, the number of bits used by each frame is itex+ptex+mv+misc. Divide by 8 to get bytes (it should be a whole number of bytes)

[edit]If you want to know how the ratecontrol works, start by looking at x264's encoder/ratecontrol.c file. That's where I got my info. Using x264farm's source will only get you more confused. :)

Thanks, I'll have a look :)

akupenguin

28th May 2008, 02:40

Basically, the number of bits used by each frame is itex+ptex+mv+misc. Divide by 8 to get bytes (it should be a whole number of bytes)
Before trying this at home, note that it only works if the 1st pass is CRF. x264's own bit-allocation algorithm gives similar results but isn't so picky.

omion

28th May 2008, 06:05

Before trying this at home, note that it only works if the 1st pass is CRF. x264's own bit-allocation algorithm gives similar results but isn't so picky.

Ah. Yes. I just realized that I actually answered a different question than what was asked. The size of the first pass frames is in itex+ptex+mv+misc (which was the question I was answering), but that does not directly correspond to the second pass frame size (which was the question asked). The stats file is processed through the functions in ratecontrol.c before the optimal second pass frame sizes are known.

Even after processing by the ratecontrol functions, there is no guarantee that the frames will actually be that size. In fact, since x264 can't directly set the number of bits used in a frame, it will always be a little off.

taemun

28th May 2008, 11:42

Adub

28th May 2008, 19:56

6> Given that x264 is highly multi-threaded is there any reason to have more than one instance of x264 running on every agent.

I was wondering about this too. Wouldn't it be faster to just let x264 run multithreaded?

foxyshadis

28th May 2008, 23:27

I was wondering about this too. Wouldn't it be faster to just let x264 run multithreaded?

Depends on if the input is bottlenecked or not. (Combine MT+x264 threading and you get better results.)

On the other hand, why does it need so much ram at all?! Graphedit can play back the full source in question with 110MB of RAM usage, why should a frame-server need any more than that?

Avisynth caches. You can use setmemorymax to control that, since avisynth caching ignores multiple instances. (This is even worse on 2.58, where default is half the available system memory, not one quarter. If they all initialize at once, they all see the same available memory.) Even if frames are never needed again, they'll sit in cache until their number comes up when the maximum memory limit is reached. Then it just stays at that limit until you're done encoding, mostly.

omion

28th May 2008, 23:49

Hey all.

I'm new to x264 encoding in general, (have worked out what most of the buttons in MeGUI etc do), and given that I have several computers lying around here, I thought that there should be a relatively easy way to get x264 to run over all of them at once.

Hence, found this project.

I've had it running, although I'd have to admit that a few things aren't clear to me:

1> Unless I specify a rate control method in the -1 "blah blah", it doesn't do anything at all.... was this intended? What value should I be using in here for a good quality 720p rip (is --qp 26 sufficient?).
That is an x264 issue. If you don't specify the ratecontrol method with x264, it will just fail with a helpful warning. The problem with x264farm is that the helpful warning isn't actually displayed...

I generally use a lower setting (--qp 22 or something) but do a few tests to see what looks good for you.

2> I'd like a little more control over the IP that the controller and agents bind to ... I have multiple IP addresses, that will go differing routes to get to the clients. One of those interfaces is gig, one is 100M. I'd rather be able to force it to go the gig path. This may be related to my inability to work out the networking parameters in the controller - the first time I tried I couldn't get anything aside from adhoc working - this suits my purposes fine, as you can kill the processes remotely, but adhoc can't specify the controller IP to be bound to.

Currently it's set to bind to any IP address, since I didn't need the ability to change it. That feature would be nice for some people, though, so I'll add it in. However, it's probably not going to get into the 2-pass version of x264farm any time soon, as I'm mainly working on the 1-pass version now.

3> I'd like to be able to change the agents list/the number of x264 instances per agent whilst the controller is running. You can sort of do this by running in adhoc, and just starting/closing/restarting the agent whenever you want to change something, but that only works (for me) if you can force the IP that the adhoc controller is running on.
I have (pretty buggy) support for that in the 1-pass version. It does still need you to restart the agent every time you want to change the settings, but it generally works. In non-adhoc mode for the 2-pass version, the agent list is defined immutably in the beginning of the encode. Changing the way that works would be fairly difficult.

4> What is AVS2YUV doing here? I presently have three instances of x264 running over two computers (one on controller, ~500MB/ram, two remote, ~230MB/ram each), and the controller also has three avs2yuv's running:
One of 766MB (no CPU-time, unless an agent recently finished a block).
Two of 550MB (3~13% CPU-time). Presumably these are frameserving across the network (controller based encoding).
[as I've been typing this, both of the 550MB processes have died, and two new ones of around 350MB have come up]

Reason I don't understand, is because x264 can handle an .avs input directly, why is there a need for a frame server here? (proviso: during agent-based encoding)
On the other hand, why does it need so much ram at all?! Graphedit can play back the full source in question with 110MB of RAM usage, why should a frame-server need any more than that?

Any controller-based encode needs to start up its own instance of avs2yuv to render the AVS file. There is also one extra avs2yuv which the controller consults for finding scene cuts. The memory usage is always a bit large by default (since Avisynth is set up to cache everything it gets its hands on) but you can limit that by using SetMemoryMax(...) in the AVS file. You are correct that frameserving should use very little RAM - every output frame is used exactly once, and in order. Therefore you should see no performance decrease by limiting the amount of memory Avisynth uses.

5> I can't seem to get agent-based encoding working, the agent throws the error:
Presumably, the AVS component is working (it is AVS throwing an error about the .grf file), and graphedit maybe doesn't like working across a Windows File Share...? U:\ is just mapped to \\10.0.0.1\u$

I can't really help you there. I don't know the inner workings of graphedit or MeGUI... Generally errors like that are due to some path being hard-coded in the AVS (or an included file), but since you have the agent's U:\ mapped to the controller's U:\, I don't really know.

6> Given that x264 is highly multi-threaded is there any reason to have more than one instance of x264 running on every agent. Aside from the small delay that occurs when dumping back the stats file, or the resulting .mkv files in the second pass. IF you're worried about that delay in the second pass, I can see why you might use pad=1; number=1. (btw, why not simplify that into, numberFirstPass=1; numberSecondPass=2?)
I honestly don't know how they compare. When x264 first added threading support I tested it out, and using multiple single-threaded x264 processes worked a little faster. But then the threading model was completely changed around and it looks like it's quite a bit better now than it used to be. Test it out and report back! (oh, and use "--threads auto", or it will force all agents to use the same number of threads, which is probably not what you want;))

foxyshadis

29th May 2008, 06:25

5> I can't seem to get agent-based encoding working, the agent throws the error:
Presumably, the AVS component is working (it is AVS throwing an error about the .grf file), and graphedit maybe doesn't like working across a Windows File Share...? U:\ is just mapped to \\10.0.0.1\u$
Come to think about it, this is probably because you don't have all of the exact same directshow filters registered on every agent as the controller. Filters are specified by CLSID in the .grf, so even different versions of the same filter may not be picked up.

taemun

29th May 2008, 07:02

Come to think about it, this is probably because you don't have all of the exact same directshow filters registered on every agent as the controller. Filters are specified by CLSID in the .grf, so even different versions of the same filter may not be picked up.

:D
Yeah I came to that conclusion late last night. CoreAVC was only installed on the controller, so of course it failed. Pity I didn't think about that at the time (after trying to dig through the .grf with notepad :P). I'd forgotten that I had used something a little more exotic when I made the graph.

That is an x264 issue. If you don't specify the ratecontrol method with x264, it will just fail with a helpful warning. The problem with x264farm is that the helpful warning isn't actually displayed...

Yeah I did see that helpful warning, was more wondering whether I was doing something wrong, because the FAQ (http://omion.dyndns.org/x264farm/x264farm.html) didn't mention it.

That feature would be nice for some people, though, so I'll add it in.

Thanks, that'd be great.

There is also one extra avs2yuv which the controller consults for finding scene cuts.

That would be the one that had 3MB in RAM and ~750MB in pagefile at the end :)

Not to tell you your job or anything, but shouldn't that instance be able to be closed after the first pass is complete? (ie after all the scene cuts have been found)

On another note, why is it that the x264 builds used by x264farm have to be made in MSVC? (ie not from x264.nl)

And: what is the newest MSVC build. The one posted a couple pages back by SpAwN_gUy called x264.819.msvc2005.modified.exe, seems to crash quite a bit (I think its when x264 has too low of a bitrate, and normally it would throw an error ... but instead, this build just crashes and then Windows sits and waits for user input - ie to close the "this program has crashed" dialog).

Cheers,
t

omion

29th May 2008, 07:43

That would be the one that had 3MB in RAM and ~750MB in pagefile at the end :)

Not to tell you your job or anything, but shouldn't that instance be able to be closed after the first pass is complete? (ie after all the scene cuts have been found)

That would certainly make the most sense, wouldn't it?:p The only real reason I didn't do that is that I had abstracted away the avs2yuv process so far that it hid all the parts that I could have closed. Rather than attempting to do that, I decided instead to try and get rid of the splitter altogether. After much hacking away, I finally decided to do a ground-up rewrite to the entire program, which turned into x264farm-sp (which doesn't have a splitter). The 2-pass x264farm has since fallen by the wayside.

On another note, why is it that the x264 builds used by x264farm have to be made in MSVC? (ie not from x264.nl)

That is due to an annoying bug that x264 will open the stdin handle in text mode, meaning that it will interpret ASCII character 0x1A as an EOF marker and stop encoding. (it also turns 0x0D0A into 0x0D) MSVC builds don't have this issue, though, as there are a few MSVC-specific lines in x264 to switch to binary.

This is only a problem in controller-based mode, though. You may use anything you want if you ensure that the computer is using agent-based encoding.

(Note that it's not really a bug in x264; I think stdin and stdout are usually opened in text-mode in Windows. It's just a bit odd that x264 expects binary data to pass through them...)

It should be an easy fix for making it work in cygwin/msys, but I really don't know enough C to say for sure.

And: what is the newest MSVC build. The one posted a couple pages back by SpAwN_gUy called x264.819.msvc2005.modified.exe, seems to crash quite a bit (I think its when x264 has too low of a bitrate, and normally it would throw an error ... but instead, this build just crashes and then Windows sits and waits for user input - ie to close the "this program has crashed" dialog).

Cheers,
t
I'm not really sure what the most current version of MSVC x264 is. I gave up making my own builds a long time ago, since x264 is really only tested against GCC, and some of the changes would break MSVC for long periods of time. Not knowing a lot of C, I was clueless to fix it in these cases. Since then, I have had no need for controller-based encoding (on Windows boxes), so I haven't kept up with the progress. I'll have to defer to SpAwN_gUy to answer that question.

If you can get agent-based encoding up and running, then you can bypass this restriction. That would probably be the best course of action right now. (if the CoreAVC fix made agent-based encoding work, then you can use anything you want)

SpAwN_gUy

2nd June 2008, 11:58

i've just moved to the new PCbox ... so i've lost the tracks of the threads...

okay, since b0bor started to post patched sources (damn git..) it will not be too hard to make MSVC builds ...

good news are: meGUI-x264farm-mod (ma-a-n... this is quite a huge name...) is still indev.. i've moved 80% of code to the latest (like two weeks ago) meGUI revision... so.. some may wait for a new version soon..

/*offtop... as far as i'm learning bash and perl (and doing some fancy scripts..).. C# is my "back to the reality" thing... so more to a hobby.. but i'll catchUP :) ... and i'm on Vista x64 now..

Adub

2nd June 2008, 22:03

Good to hear from you again. As you know, looking forward to the mod!! Thanks again.

thebigfatgeek

7th June 2008, 23:19

Hi, I have just installed the agents and the controllers, modified th config.xml files and I keep on getting the same result. I cannot see the error, or that anything is happening (except some processor activity on controller.exe and avs2yuv, but after 0:46 seconds, it stops, it does not complain (which is nice) but it also does not work (which is not nice)

This is the output from the out-dump.txt file:

2008-06-07~23:09:23.53 0% done (0 / 49544) at 0.00 FPS
2008-06-07~23:09:23.53 Last updated: 0:45
2008-06-07~23:09:23.53 ----...----________________________________...............-----...............
2008-06-07~23:09:23.53 Agent | FPS | Description
2008-06-07~23:09:23.53 Recent errors:
2008-06-07~23:09:23.53 ~
2008-06-07~23:09:23.53 ~
2008-06-07~23:09:23.53 ~
2008-06-07~23:09:23.53 ~
2008-06-07~23:09:23.53 SPLITTER MUTEX unlocking range_list_mutex and yielding
2008-06-07~23:09:23.53 SPLITTER MUTEX sleeping for 1/10 second
2008-06-07~23:09:23.64 SPLITTER MUTEX locked range_list_mutex after yielding
2008-06-07~23:09:23.64 e 0
2008-06-07~23:09:23.64 f 0
2008-06-07~23:09:23.64 q 3
2008-06-07~23:09:23.64 w 0
2008-06-07~23:09:23.64 es 3
2008-06-07~23:09:23.64 fs 0
2008-06-07~23:09:23.64 qs 8
2008-06-07~23:09:23.64 ws 0
2008-06-07~23:09:23.64 Splitter: current batch length is 2116
2008-06-07~23:09:23.64 Splitter: try_queue_after_full_or_working
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 batch length: 2116
2008-06-07~23:09:23.64 Splitter: try_extend_small_thresh_queue
2008-06-07~23:09:23.64 extend_small_thresh_queue found no ranges of sufficient patheticosity
2008-06-07~23:09:23.64 Splitter: try_split_largest_range
2008-06-07~23:09:23.64 Splitter: wait_on_full
2008-06-07~23:09:23.64 SPLITTER MUTEX broadcasting range_list_empty_condition
2008-06-07~23:09:23.64 0 @ INFINITY
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 2502 @ 7.56
2008-06-07~23:09:23.64 Empty
2008-06-07~23:09:23.64 4607 @ 10.63
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 6857 @ 4.69
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 9282 @ 2.17
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 11709 @ 7.17
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 14176 @ 2.42
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 16791 @ 6.33
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 19347 @ 7.58
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 22022 @ 9.34
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 24696 @ 4.79
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 27246 @ 3.65
2008-06-07~23:09:23.64 Empty
2008-06-07~23:09:23.64 37171 @ 11.85
2008-06-07~23:09:23.64 Queued
2008-06-07~23:09:23.64 39525 @ 6.16
2008-06-07~23:09:23.64 Empty
2008-06-07~23:09:23.64 49544 @ INFINITY
2008-06-07~23:09:23.64 0% done (0 / 49544) at 0.00 FPS
2008-06-07~23:09:23.64 Last updated: 0:45
2008-06-07~23:09:23.64 ----...----________________________________...............-----...............
2008-06-07~23:09:23.64 Agent | FPS | Description
2008-06-07~23:09:23.64 Recent errors:
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 SPLITTER MUTEX waiting on range_list_full_condition; unlocking range_list_mutex

and my controller was started with:

Controller>controller -B 4500kbps --first "--bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --me dia --me range 12 --progress --no-psnr" --second "--ref 3 --bframes 3 --b-pyramid --weigh tb --direct auto --filter -2,-1 --subme 6 --trellis 1 --analyse all --8x8dct --m e umh --merange 12 --progress --no-psnr" --avs "24.avs" -o "24.mkv" --batch 2500 --split 250 --preseek 5 --config config.xml

Any ideas or comments would be appreciated

Cheers

ajp_anton

16th July 2008, 15:33

Bug!

If the .avs file to encode is on a network share, it looks like "\\computer\folder"
When nice is used, it reads the "\\" as "\". It needs "\\\" to get it right, but then the .avs file won't be found and avs2yuv starts sending massive amounts of data across the network.

Apparently I can get around this by creating a .avs file that finds its inputs via the network, and copy that file onto every computer, even the local one (since it seems intelligent enough to check if the files are the same... =)

Otherwise this thing is really the best thing ever. Too bad it's not until now that I really needed it as it took only 10 mins to figure everything out and get it working.

SpAwN_gUy

17th July 2008, 09:08

If the .avs file to encode is on a network share, it looks like "\\computer\folder"
When nice is used, it reads the "\\" as "\". It needs "\\\" to get it right, but then the .avs file won't be found and avs2yuv starts sending massive amounts of data across the network. i have no problem with that ...
which nice are you using? (the one in the first post) jr the original? .. 'cause we had some problems using the original one.. and some problems using the tiny one(1st post)

currently i'm using the one in the first post... and i'm doing agent-based..

SpAwN_gUy

17th July 2008, 09:12

2008-06-07~23:09:23.64 Last updated: 0:45
2008-06-07~23:09:23.64 ----...----________________________________...............-----...............
2008-06-07~23:09:23.64 Agent | FPS | Description
2008-06-07~23:09:23.64 Recent errors:
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 ~
2008-06-07~23:09:23.64 SPLITTER MUTEX waiting on range_list_full_condition; unlocking range_list_mutex
controller haven't found any agents on the network..
check <ad-hoc> presence in controller's config..
AND try adding IP's of your agents like the old way (some of my agents are not seen some times with ad-hoc only.. but with both - it's perfect..)

SpAwN_gUy

17th July 2008, 09:18

oh.. jeah.. right.. the mod.. i need to start MSVC... and stop doing nothing...

btw.. about builds of x264... it is reported, that MSVC builds (with current revision of x264) are TOTALLY BROKEN.. and must not be used :(( ...
but the good nwes are.. that x264_mingw_stdin_patch is IN the TRUNK :) .. (yeah, yeah,.. thanks to me.. but you are welcome..) so all the builds, that are currently made.. (like on x264.nl and all the patched ones) need to be checked(at first).. and then.. well.. the must be working and compatible with controller-based x264farm encoding process..

thebigfatgeek

19th July 2008, 02:19

controller haven't found any agents on the network..
check <ad-hoc> presence in controller's config..
AND try adding IP's of your agents like the old way (some of my agents are not seen some times with ad-hoc only.. but with both - it's perfect..)

I can get it to see the agent, but I just cannot get a controller startup command that will work:

nice -n 20 "x264.exe" --bframes 3 --b-pyramid --direct auto --filter -2,-1 --s
bme 1 --analyse none --progress --no-psnr --qcomp 0.6 --cplxblur 20. --qblur 0.
--qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 25/1 --pass
1 --stats ".\temp\stats b26912.txt" -o NUL - 640x496
264 [error]: no ratecontrol method specified
264 [error]: x264_encoder_open failed