View Full Version : x264farm: distributed encoding
Pages :
1
2
3
4
5
6
7
8
9
10
11
12
[
13]
14
Wishbringer
19th July 2008, 10:44
Controller>controller -B 4500kbps --first "--bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --me dia --me range 12 --progress --no-psnr" --second "--ref 3 --bframes 3 --b-pyramid --weigh tb --direct auto --filter -2,-1 --subme 6 --trellis 1 --analyse all --8x8dct --m e umh --merange 12 --progress --no-psnr" --avs "24.avs" -o "24.mkv" --batch 2500 --split 250 --preseek 5 --config config.xml
You did set up the final bitrate for the second pass, but
264 [error]: no ratecontrol method specified
look at first pass string , there is no ratecontrol set. I would add something like --crf 18 into first pass string
see: http://forum.doom9.org/showpost.php?p=1143312&postcount=589 some posts before.
SpAwN_gUy
5th August 2008, 10:51
hi, guys, i've been asked to help with x264farm setup...
and i'm on the new pcBox ... so i had some time.. to just test everything from the scratch or something like this...
http://www.ljplus.ru/img4/s/p/spawn_guy/th_x264farm-vista-x64-doubleAgent-gcc_928.jpg
fullPicture is here (http://www.ljplus.ru/img4/s/p/spawn_guy/x264farm-vista-x64-doubleAgent-gcc_928.jpg)
the Picture shows win Vista SP1 x64 ... with two running agents.
one from commandline, - this one is doing controller-based
second started over the network through my agentGUI. - this one is agent-based
they both use nice at 20,
and mingw compiled x264 x86 r928 from b0bor's site .. (plain git)
no x264x64 (i havent found one x64 for now)
aviSynth 258.. but x86
test encode uses mpeg2-progressive source, mpeg2-Telecined with YATTA, x264-source form HDTVstream using DGAVCDec..
to get all thing working i needed to (this is imporatnt) set nice's compatibility mode to winXP-SP2 ... otherwise the whole thig was not working.. (but only for agents...)
and.. UNBLOK controller and agents in winFirewall !!!
and that's just it :) ...
any questions are welcome ....
(i guess, i should better show the meGUI-Mod.. but it is still unusable... (well, the new version))
controller's config.xml
<config>
<temp>controller-temp</temp>
<!-- Some directory for storing temp data -->
<adhoc controller="40700" agent="40701"/>
<agents>
</agents>
</config>
agent's config (the command-line controller-based one):
<config>
<temp>temp</temp>
<port from="40700" to="40701"/>
<x264>x264.exe</x264>
<nice>20</nice>
<adhoc controller="40700" agent="40701"/>
<number>1</number>
<agentpipe>0</agentpipe>
<compression>
<type>1</type>
<type>0</type>
</compression>
</config>
angent's config (the one that is common and is started over the network with my agentGUI (it can be started the same way using "\\myPC\x264farm\!AGENT\agent.exe --config \\myPC\x264farm\!AGENT\config.xml")):
<config>
<temp>temp</temp>
<port from="40700" to="40701"/>
<x264>\\myPC\x264farm\!AGENT\x264.exe</x264>
<nice>20</nice>
<bases>
<base>\\myPC\x264farm-2\</base>
<base>\\myPC\x264farm\</base>
</bases>
<adhoc controller="40700" agent="40701"/>
<number>1</number>
<agentpipe>0</agentpipe>
<compression>
<type>1</type>
<type>0</type>
</compression>
</config>
theese are the internals of test.avs used:
SetMemoryMax(64)
# PLUGINS
LoadPlugin("\\myPC\x264farm\TEST\filters\DGDecode.dll")
LoadPlugin("\\myPC\x264farm\TEST\filters\FluxSmooth.dll")
LoadPlugin("\\myPC\x264farm\TEST\filters\DGAVCDecode.dll")
LoadPlugin("\\myPC\x264farm\!filters\TDeint.dll")
LoadPlugin("\\myPC\x264farm\!filters\TIVTC.dll")
function mpeg2(){
clip1=mpeg2source(".\mpeg2\test.d2v", idct=0)
clip1=crop(clip1,14,82,696,412)
return LanczosResize(clip1,640,480)
}
function mpeg2anime(){
d2vpath = ".\mpeg2-anime\yatta-test.d2v"
clip1=mpeg2source(d2vpath, idct=0)
deint = tdeint(clip1)
clip1 = tfm(clip1, d2v=d2vpath, clip2=deint)
return LanczosResize(clip1,640,480).AssumeFPS(25).Trim(400, 3400)
}
function ColBars() {
return ColorBars(640,480).AssumeFPS(25).ConvertToYV12().KillAudio.Trim(0,1000)
}
function h264() {
clip2=AVCSource(".\h264\hgttg-test.dga", deblock=false)
return clip2.AssumeFPS(25).ConvertToYV12().KillAudio.LanczosResize(640,480)
}
mpeg2().Info.Overlay(Version, x=200, y=400) + ColBars().Info.Overlay(Version, x=200, y=400) + h264().Info.Overlay(Version, x=200, y=400) + mpeg2anime().Info.Overlay(Version)
the is: E:\DVDRIP\x264farm\TEST\mpeg2\test.d2v internals (the .vob is located just near E:\DVDRIP\x264farm\TEST\mpeg2\VTS_01_8.VOB)
DGIndexProjectFile16
1
VTS_01_8.VOB
Stream_Type=1
MPEG_Type=2
iDCT_Algorithm=5
YUVRGB_Scale=1
Luminance_Filter=0,0
Clipping=0,0,0,0
Aspect_Ratio=16:9
Picture_Size=720x576
Field_Operation=0
Frame_Rate=25000 (25/1)
Location=0,0,0,b9de
900 5 0 20480 0 7 18 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 e2
d00 5 0 258048 0 7 18 d2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
.......
this is E:\DVDRIP\x264farm\TEST\h264\hgttg-test.dga
the only part i har to change manually is path to libavcodec (this is not implemented to be relative-path .. i had to change it to network-path)
DGAVCIndexFile5
hgttg-test.h264
\\myPC\x264farm\TEST\filters\libavcodec.dll
FO 2
LPOC 0
SPS 0
PPS 28
this is E:\DVDRIP\x264farm\TEST\mpeg2-anime\yatta-test.d2v (the one that is manually inverse-Telecined with YATTA ( ivtc.org ))
DGIndexProjectFile16
1
VTS_01_5.VOB
Stream_Type=1
MPEG_Type=2
iDCT_Algorithm=5
YUVRGB_Scale=1
Luminance_Filter=0,0
Clipping=0,0,0,0
Aspect_Ratio=4:3
Picture_Size=720x480
Field_Operation=0
Frame_Rate=29970 (30000/1001)
Location=0,0,0,44837
900 5 0 2048 0 5 2 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2 b2 b2 a2
...
aviSynth filters are located here
localPath: E:\DVDRIP\x264farm\TEST\filters\
networkPath: \\myPC\x264farm\TEST\filters
basically.. my setUP is to use network-share:
localPath: E:\DVDRIP\x264farm
networkPath: \\myPC\x264farm
for my agentGUI i use common agent setUP located here:
local: E:\DVDRIP\x264farm\!AGENT
network: \\myPC\x264farm\!AGENT
that's i folk :) ... it's pretty simple...
the some kind of minimal setUP can be obtained form here (http://rapidshare.com/files/134974974/x264farm-minimal.rar)
winAMP plays: [Apex - Same Old Blues]:[Drum & Bass-2008]:-:[5:40s]:[320-44KHz-stereo]:[MP3]
BTW: if you find any confidential data in things, that i wrote here .. please.. DON'T USE IT.. and tell me about it..
some mistakes can be.. that "myPC" real network name is "SC0151" remember that...
Jabroni
15th September 2008, 23:05
Im trying to get this working with a newer X264 build (techouse builds), however Im getting some probs. I have been able to make it process the frames, the agent doesnt display any errors at all, neither on the controller, but the % of what is done never increases (it seems like the agent isnt returning the encoded info)
Any place what I should be looking into ??
:thanks:
omion
15th September 2008, 23:24
Im trying to get this working with a newer X264 build (techouse builds), however Im getting some probs. I have been able to make it process the frames, the agent doesnt display any errors at all, neither on the controller, but the % of what is done never increases (it seems like the agent isnt returning the encoded info)
Any place what I should be looking into ??
:thanks:
I figured this may come up. x264 recently changed the stats file format, and I had an idea that would break x264farm somehow. Your description of the problem is exactly what I would expect to happen, so this is almost definitely the issue. The fix shouldn't be too hard, but I haven't built this version of x264farm in such a long time that I don't know how long it will take to start back up.
Jabroni
15th September 2008, 23:30
I figured this may come up. x264 recently changed the stats file format, and I had an idea that would break x264farm somehow. Your description of the problem is exactly what I would expect to happen, so this is almost definitely the issue. The fix shouldn't be too hard, but I haven't built this version of x264farm in such a long time that I don't know how long it will take to start back up.
Ok good.. thought this was just me being stupid on setting the new build :p Well for now i'll just stick with an old x264.exe build (the one that I had working previously)
This is the url of the custom build that I tried hxxp://techouse.project357.com/ (latest svn release) If you need any more info for debug let me know and I'll happly try to help :)
kemuri-_9
16th September 2008, 00:54
I figured this may come up. x264 recently changed the stats file format, and I had an idea that would break x264farm somehow. Your description of the problem is exactly what I would expect to happen, so this is almost definitely the issue. The fix shouldn't be too hard, but I haven't built this version of x264farm in such a long time that I don't know how long it will take to start back up.
if you really need to get back to the old stats format, you can just make a build that has that revision which affected it reversed.
keep all the other improvements and such...
SpAwN_gUy
16th September 2008, 14:42
so... we are back in bizzznisss? ;) ... cool :) ...
'cause re-encoding avchd from my sony hdr-sr12 to something reasonable at 720p... is really painful.. just "standalone"...
megui-mod update is really close :)
ajp_anton
28th September 2008, 19:29
So are we going to see a version of x264farm that supports the new stats files?
omion
29th September 2008, 16:57
So are we going to see a version of x264farm that supports the new stats files?
It'll be out eventually. The problem that I've had is that after I fixed the stats problem I noticed that the encode does not progress, so I'll have to fix that as well, and I haven't looked at the x264farm code in a long time. Part of the reason it's taken so long is that I also updated the regular-expression parser because I no longer have PCRE on my system. So it's still being worked on...
Nakaleen
8th October 2008, 02:22
Hi I am new to x264farm. Arfter some playing around I got x264farm to work over a network, also for it to work inside Staxrip.
My problem is that x264 keeps crashing on all my computers on the last section of the second pass. All it says is 'x264 as stopped working' and waits for me to close that error window, which is a pain when I leave it overnight only to find that is hasen't even finished one encode hehehe.
I have tried diffrent x264 builds, diffrent os (also crashes in the same spot on XP) a later version of Nice (ever tried to find the program 'Nice' via google, not easy hehehe) but x264 still reports an error
Here is an example of my setting (it is a modified version of Staxrip's Highprofile settings)
Controller.exe --first "--bitrate 907 --level 4.1 --bframes 3 --b-pyramid --threads auto --progress --sar 1:1 --no-psnr --no-ssim --subme 1 --partitions none --me dia" --second "--bitrate 907 --level 4.1 --ref 3 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --subme 6 --partitions all --8x8dct --trellis 1 --threads auto --progress --sar 1:1 --no-psnr --no-ssim" --output "R:\Movies\Ah My Goddess\Ah My Goddess_EncoderOutput.mkv" --avs "R:\Movies\Ah My Goddess\Ah My Goddess.avs"
I am running with Agent Encoding...
I think that all I can add, I've tried sifting through the out-dump.txt files but can't see anything that says what caused the errors.
I would LOVE for this to work as it has increased my encoding speeds from 12fps to about 30-35fps (if I keep an eye on all my computers to close down all the error boxes hehehe)
Sagekilla
8th October 2008, 02:28
Is there any chance you could actually use x264farm for crf 1-pass? I can imagine it wouldn't be much more complex, since it would be more or less identical to 1-pass.
professor_desty_nova
8th October 2008, 08:31
Is there any chance you could actually use x264farm for crf 1-pass? I can imagine it wouldn't be much more complex, since it would be more or less identical to 1-pass.
Are you talking of x264farm-sp - single-pass distributed encoding (http://forum.doom9.org/showthread.php?t=136053) ;)
SpAwN_gUy
8th October 2008, 09:21
My problem is that x264 keeps crashing on all my computers on the last section of the second pass. All it says is 'x264 as stopped working' and waits for me to close that error window, which is a pain when I leave it overnight only to find that is hasen't even finished one encode hehehe.how are you running agents?
from console or from "staxrip-agent"?
if from console the you could see x264's error.. if from satxrip - then you will not see the message...
i've had some "crashings" also... and (as i remember) i've implemented part of code to mine agentGUI to suppress those "XP-Fault-Windows" ... and i think i did the same thing to my delphi-nice-implementation (or not mine.. ).
but i still have "new-vista-error-windows" (haven't checked for new solution yet)
so.. or you kick staxrip-writer to implement "suppressors" (it's just 1 line of code on delphi) .. or you wait for new x264farm (i've told omin this issue.. long time ago).. or i'll look for my "nice.exe" somewhere here ...
"suppressors" must be implemented in a higher level application..
like staxrip-agent -then-> x264farm-agent -then-> nice.exe -then-> x264.exe
Nakaleen
11th October 2008, 10:34
Sorry for the delay. First week back from work so I haven't had time to play around with this again.
how are you running agents? from console or from "staxrip-agent"?
I don't know what you mean by 'Staxrip-agent'. Agent is running on every computer (3) from a console (started from a shortcut).
Each agent has the following 'config.xml' (execept for the <name> and <port> which is diffrent)
<config>
<temp>C:\Agent</temp>
<port from="40720" to="40729"/>
<adhoc controller="50700" agent="50701"/>
<name>Main</name>
<number>2</number>
<x264>x264.exe</x264>
<nice>10</nice>
<base></base>
<agentpipe>0</agentpipe>
<compression>
<type>0</type>
<type>1</type>
</compression>
</config>
I am using Staxrip with it's Commandline encoding option with the following....
C:\Controller\Controller.exe" --first "--bitrate %video_bitrate% --level 4.1 --bframes 3 --b-pyramid --threads auto --progress --sar 1:1 --no-psnr --no-ssim --subme 1 --partitions none --me dia" --second "--bitrate %video_bitrate% --level 4.1 --ref 3 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --subme 6 --partitions all --8x8dct --trellis 1 --threads auto --progress --sar 1:1 --no-psnr --no-ssim" --output "%encoder_out_file%" --avs "%source_dir%%target_name%.avs"
Which makes for an example....
"C:\Controller\Controller.exe" --first "--bitrate 920 --level 4.1 --bframes 3 --b-pyramid --threads auto --progress --sar 1:1 --no-psnr --no-ssim --subme 1 --partitions none --me dia" --second "--bitrate 920 --level 4.1 --ref 3 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --subme 6 --partitions all --8x8dct --trellis 1 --threads auto --progress --sar 1:1 --no-psnr --no-ssim" --output "R:\Vandread\Vandread s01-e01\Vandread s01-e01_EncoderOutput.mkv" --avs "R:\Vandread\Vandread s01-e01\Vandread s01-e01.avs"
Drive R:\ is a shared drive mapped at every computer to R
if from console the you could see x264's error.. if from satxrip - then you will not see the message...
Looking at the out-dump.txt this is the error reported.
2008-10-11~19:08:13.10 218 2nd pass frames 244 to 273
2008-10-11~19:08:13.18 218 done recieving stats
2008-10-11~19:08:13.20 Testing file "R:\Vandread\Vandread s01-e01\Vandread s01-e01.avs": FOUND!
2008-10-11~19:08:13.20 218 agent-based encoding
2008-10-11~19:08:13.21 218 doing the following:
nice -n 10 "x264.exe" --bitrate 890 --level 4.1 --ref 3 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --subme 6 --partitions all --8x8dct --trellis 1 --threads auto --progress --sar 1:1 --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --bitrate 11 --fps 25/1 --pass 3 --stats "C:\Agent\stats 94473f.txt" --seek 244 --frames 30 -o "C:\Agent\output f8b890.mkv" "R:\Vandread\Vandread s01-e01\Vandread s01-e01.avs"
2008-10-11~19:08:28.65 218 exited -1073741819
2008-10-11~19:08:28.67 Thread received exception "Failure(\"ERROR: nonzero return code from x264 (-1073741819)\")"
2008-10-11~19:08:28.75 closing socket
2008-10-11~19:08:28.75 socket closed
so.. or you kick staxrip-writer to implement "suppressors" (it's just 1 line of code on delphi) .. or you wait for new x264farm (i've told omin this issue.. long time ago).. or i'll look for my "nice.exe" somewhere here ...
"suppressors" must be implemented in a higher level application..
like staxrip-agent -then-> x264farm-agent -then-> nice.exe -then-> x264.exe
delphi is WAY out of my leage, as well as most programing (haven't done any since C++ 10 years ago hehehe) so there is nothing I can do in at area hehehehe
SpAwN_gUy
13th October 2008, 09:31
Agent is running on every computer (3) from a console (started from a shortcut).
Looking at the out-dump.txt this is the error reported. out-dump.txt will not give you the resuls i mean..
you should look on the console-window, where agent.exe runs.
'cause x264.exe outputs information direct to console. and out-dump.txt does not have this information. while "console window" still has the information.
Nakaleen
2nd December 2008, 23:16
out-dump.txt will not give you the resuls i mean..
you should look on the console-window, where agent.exe runs.
'cause x264.exe outputs information direct to console. and out-dump.txt does not have this information. while "console window" still has the information.
Desided to give this program a try again....
And again it is giving me a head ache from all the keyboard slamming.
3 computers, 3 diffrent setups, a mixture of XP, vista, Intel, amd, laptops etc... no matter what setup they all crashes at some stage. This is very annoying as I set up to encode 5 videos overnight and wake up to see it crashed at the end of the first one.
I copied this from the last render off one of the agents, that caused a crash.
815 2nd pass frames 35531 to 35531
815 done recieving stats Testing file "R:\Read Or Die\Read Or Die S01E02\Read Or Die S01E02.avs": FOUND!
815 agent-based encoding
815 doing the following:
nice -n 10 "x264" --bitrate 917 --level 4.1 --ref 3 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --subme 6 --partitions all --8x8dct --trellis 1 --threads auto --progress --sar 1:1 --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --bitrate 1853 --fps 25/1 --pass 3 --stats "C:\Agent\stats 24b0cf.txt" --seek 35531 --frames 1 -o "C:\Agent\output b6babc.mkv" "R:\Read Or Die\Read Or Die S01E02\Read Or Die S01E02.avs"
avis [info]: 640x480 @ 25.00 fps (35532 frames)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 Cache64
encoded frames: 1/1 (100.0%), 4.02 fps, eta 0:00:00
815 exited -1073741819
Thread received exception "Failure(\"ERROR: nonzero return code from x264 (-1073741819)\")"
closing socket
socket closed
Thread 822 killed on uncaught exception Unix.Unix_error(40, "send", "")
It is not much diffrent then what is in the log files. So I am stumped that is causing the error. Searching for error code -1073741819 for x264 found nothing. Maybe it is one of my x264 setting... but I can't find any info on what setting could crash all my systems.
Ichling
3rd December 2008, 00:00
Hi Nakaleen,
can you give me the exact options that you used to start the controller with? I was running in similar problems with my first tries at x264farm(-sp) and they, in the end, where all related to the x264 settings i've used.
Jabroni
1st February 2009, 19:14
So has anyone done any custom build of x264 thats a bit newer to current SVN release ? I want to make DXVA compatible encodes, and I think I need some parameters that the old x264s doesnt have (or at least the one that works with x264farm).
SpaWnGuY: Are you still working on your megui-mod? I accidently 'upgraded' the megui install, and now it removed all the x264farm stuff, and I cant find the installer of it =\ Would you mind posting it again please?
:thanks:
Pulp Catalyst
12th February 2009, 16:28
hopefully when BD Re-Builder is released someone will design a Farm program like this one
http://www.afterdawn.com/guides/archive/dvd_rebuilder_tutorial_advanced_page_8.cfm
RB-Farm was brilliant and worked great, why you ask, because the way DVD Re-builder pulls apart the DVD,
hopefully BD Re-builder will do the same,
Jabroni
12th April 2009, 19:38
Was wondering if there is going to be any update to this project and/or MeGUI. Would love to distribute all my new encodes with x264farm (im using parameters used on newer x264.exe build, and for some reason they make the x264farm clients crash)
:thanks:
Allbestmessage
15th April 2009, 10:30
There is a command line encoder profile you could use.
Breakthrough
24th July 2009, 18:47
Hello, I have a few questions...
What build of x264 do you all reccommend to use with x264farm? Or at least, what was the final build before the revision to the .stats file was committed?
Also, does the server computer also have the ability to render? Could I theoretically run a controller AND an agent on the same PC?
Thanks!
nst6563
28th July 2009, 19:50
I'm itching to get this working as designed...however it seems I've either got something setup wrong, or something is setup wrong :p
To start, I've got the controller running on a 3Ghz Core2Quad with 6GB ram. I have another machine that is connected via 100mb switch which is a core2duo w/4gb ram.
I've got the config setup for adhoc, and the controller and client see each other - that's not the problem.
The problem I think I've run into is the speed. I was hoping it would be faster than standard encoding with something like ripbot or handbrake...however I get about half the speed compared to those gui-based encoders.
I've tried the controller-based encoding as well as the client-based encoding. I don't remember if I got the client based encoding to work correctly tho... the way I read and figured out was that each client needs access to the sources being encoded right? I made a mapped drive on the machines, and pointed the config to that and it seemed to work ok.
I have 5 machines with a total of 13 cores that I could put towards something like this...I'd love to see it scream though an encode...what could be my limiting factor other than the 100mb switch (trying to get a 1000 but my current 1000 switch shot craps...thanks Dlink)
any plans of something like a gui for the controller setup? Drag a bunch of files to encode to the window (or provide a path with the files to encode), answer some questions and let it go. maybe launch the agents as a windows service?
iplmedia
15th January 2010, 07:17
I'm looking to bring this code up to date with the latest x264 and to bring it to maturity. If anyone, Ideally the author, is able to help in this please let me know.
creamyhorror
15th January 2010, 17:10
I'm looking to bring this code up to date with the latest x264 and to bring it to maturity. If anyone, Ideally the author, is able to help in this please let me know.
Don't forget the single-pass variant: x264farm-sp - single-pass distributed encoding (http://forum.doom9.org/showthread.php?t=136053), which was last modified in 2008 I think.
creamyhorror
18th January 2010, 16:54
I'm looking to bring this code up to date with the latest x264 and to bring it to maturity. If anyone, Ideally the author, is able to help in this please let me know.
Another guy is interested in developing distributed encoding for x264: Distributed Encoding with x264 (http://forum.doom9.org/showthread.php?p=1364155#post1364155)
bnshrdr
18th January 2010, 21:18
Another guy is interested in developing distributed encoding for x264: Distributed Encoding with x264 (http://forum.doom9.org/showthread.php?p=1364155#post1364155)
I really do like the idea and am fully capable of the actual design of the program. The only thing I truly lack is knowledge of x264. Sure I use it for all of my encodes, but I don't know enough about it to write pretty much a wrapper program to utilize it.
Currently I have 2 computers here at my home encoding the same video file, but some of the ways I am achieving this are either ugly or impractical. Also I don't really know how to truly handle a multi-pass scenario.
I just need info on how to do faster seeking, like using an index. I know of DGAVCDec but I don't know how to put it to good use.
I also need a way to get the frame count of the source before processing the job, but my knowledge of the AVISynth API is 0.
I am also welcoming any ideas about the whole design of this program.
iplmedia
19th January 2010, 22:17
I really do like the idea and am fully capable of the actual design of the program. The only thing I truly lack is knowledge of x264.
If am very familiar with x264 just not how to wrap it! x264farm has many benefits to it by dividing the video into GOP, however, it decompresses it to ray YUV before compression (From what I understand) and this is done in the first pass that only 1 machine performs. Thus greatly slowing down the encoding process. I would love to discuss the possibility of working on this project if you like. Send me an email and we can discuss this.
creamyhorror
20th January 2010, 00:58
Great, hope you guys get together to start a project. You might want to hang out in #x264dev on Freenode IRC to ask the x264 devs questions.
iplmedia
20th January 2010, 21:47
I have worked with and am working with one of the x264 developers already on a different project. They are a great team of developers!
bnshrdr
21st January 2010, 05:36
As much as I'd like to start a little project, I really don't have time. During the school year I always nibble little bits away from my project goals. I'm kind of just a hobby programmer, but being a full time student limits the time I can contribute to any sort of project. Thanks though for the offer and I appreciate your support. I know where I'll come when I have problems :P.
popper
21st January 2010, 14:39
the Very first thing you should perhaps do is totally do away with a fixed x264 version requirement in any code so as not to run up against the code updating being dropped as happened with farm for SO long...
it might be a very good idea to start by parsing a current x264 --fullhelp at runtime and populating your input/output variables from that parse, so that as fullhelp grows over weeks,months or years your code can take advantage of any new options that come along later and still work fine when things change.
as it is today the general feeling is that always using the constant quality option CRF= as your base is the default way to go.
x264farm while a good idea always bugged me in the way it doesnt seperate out all the different parts of the operations code into self contained stand alone scripts, so making it really hard to add new or just different ways of doing the same thing for a given situation,as a plug-in type way to work, so that could be made far cleaner if YOUR really serious about this rework.
you might want to add a new better client/server part but keep all the other bits as they were for instance, or you might want to add DHT for the auto what Hw's available to our IP cluster at this time, and who's ready to take another job, send a DHT message and add and/or act to the 'message based' actions que(s) etc... the options are endless with a little thought and an open self contained fully commented script for one action, or were needed for simpicity a set of related fine grained actions etc.
it should also be considered that current x264 also takes many input file types stand alone now OC so thats a good thing, and sooner or later it will also allow audio parsing/re-encoding to a given container too, so making the whole updated x264farm framework far easier and better later IF YOU JUST PARSE a current x264 fullhelp output or ask DS to provide a better way perhaps if your serious about distributed encoding.
for instance how might a future x264 farm make easiest use of things like this guys http://forum.doom9.org/showthread.php?t=152232 with HPC cluster (750 cores)...
people will always want to use the latest x264 so you had better find an easy way to always auto include/use it with a new x264farm framework without braking the farm....if you want people to use it and give lots of thanks and perhaps even get involved for years to come ;0)
and preferably cross platform usable too, put it infact the whole turnkey package on a generic slax liveCD http://www.slax.org (WITH DEFAULT ready setup pxe server LAN booting AS STANDARD BTW) AND boot any hardware thats in the LAN ANYTIME.
just a few thoughts to consider anyway.
Karyudo
22nd January 2010, 17:53
Good timing! I just started researching how to build some kind of x264 render farm, and here I see new interest after what looks to be a year or more of downtime and benign neglect. Nice!
I can offer almost zip in the technical department, but I do have a home network with four (soon to be five?) computers that could conceivably all run DGDecNV on at least Core2Duo processors, so maybe I'd be able to help with beta testing (when things progress that far, I mean).
Or, if that's no big help, then at least let me add my vote for this idea as a project that I think would be powerful and useful!
iplmedia
22nd January 2010, 22:05
Thank you for your thoughts on this project. I am currently planning out the best architecture for this project trying to utilize the best possible means to accomplish the best possible product. I agree with future compatibility as well as cross-platform. I currently work solely on a mac and would like to see it work on that but understand their are many other users in the *nix and windows world. The concept of a liveCD for the cluster is very familiar to the Linux world (Clusterknoppix etc).
I am considering using OpenMPI and dividing the video into GOP's for distribution so regardless of how many videos you have in queue the entire cluster will be utilized. I realize that their are inherent flaws and considerations but my goal would be very fast cluster detection, minimal bottle-necking and intelligent distribution amongst all available resources with fault-detection and load balancing.
It's a tall order but in my opinion much needed and sought after.
Please continue to post your thoughts/ideas/suggestions.
Thanks
kemuri-_9
23rd January 2010, 03:37
nitpick: OpenMPI is only an implementation of MPI-2, you would be using the MPI-2 API if using OpenMPI as a test implementation.
I had worked somewhat on an MPI-2 patch (was using the MPICH2 implementation myself for windows) nearly a year ago for x264 but stopped eventually when i realized that it would require a fairly large restructure.
I had also planned on GOP distribution among the cluster and was coding along that guideline.
it would probably be easier to implement now utilizing threaded lookahead that was committed some moths ago.
magnatique
24th January 2010, 19:05
when this is up to a testing phase, I would be happy to give the farm some testing grounds as we have core2, quads i7s at the office on our gigabit lan, with some on XP / win7 so it should give a good testing ground!
Phil
iplmedia
24th January 2010, 23:14
Hello all,
Thank you for your input. I am continuing to research and gather the best possible method to accomplish this and your input helps eliminate dead-ends and un-needed wastes of time etc. So please keep them coming. In regards to the MPI-2 implementation I still see it as the best method as well as the GOP. What were some road-blocks you ran into with using MPI and would you suggest a different method or is it still be best method just that it requires a lot of work. I am willing to invest the work/time/investment that it would take to accomplish this I just want to make sure I'm heading in the right direction. Would you be willing to send me the work that you have started already on the x264 MPI implementation? Any thoughts and information you can pass my way would be greatly beneficial. Cheers
Aaron
kemuri-_9
25th January 2010, 01:56
hmm.. having to pick my brain as to the thought processes i had back when i was working on it...
the MPI-2 API has some functionality to determine rank in the operational cluster (along with the size of the cluster) and i used this to define the following:
a node with a rank of zero would indicate the master node - that is the node that is controlling source decoding to send and receive frame data back from slave nodes for writing the final encoded video
otherwise a node has a rank of nonzero and would be considered a slave node.
A) Where to decide where to transfer frames between master and slave nodes
When i was working on this i had issues on where I was going to do this and was a major factor in stopping work.
A1) from working on the threaded lookahead patch,
I would think it would be easiest location for transfer frames from the master node to the slave node.
A2) as for the reverse.... don't recall thinking about that much... thoughts now would have it either in encoder_frame_end or in x264cli
B) what requires transferring:
originally when i was working on this there was only 1 frame type, but now with there being 2, this changes up a little bit.
B1) fdec x264_frame_t
frames that have been slicetype decided, would be transferred from master to slave for encoding.
B2) fenc x264_frame_t?
frames that have been encoded and QP decided, would be transferred from slaves to master.
this is a bit tricky as x264_frame_ts have a lot of data that would be unncessary at the master slave as all it would really want is the resulting x264_picture_t.
though if you only do decide to send back the x264_picture_t you would have to special case the --dump-yuv option in some fashion.
B3) stats
information acquired while processing frames,
involving bit costs and other information used at encoder_close and in stat file writing,
would need to be synced between master and slave node (i didn't get to this part so not sure what the best way of doing it would be).
B4) x264_param_t
the parameters to init the encoding engine with,
I had ambiguous thought patterns on how to handle this,
but i think my last choice was to init the master node with the param_t given on the cli and then send a possibly altered param_t to slaves to init with.
C) Gotchas i remember
C1) data sending.
transferring structures is eloquent in MPI-2 especially when they have pointed to data.
this required creating an extra send buffer and `packing` along data in a struct referred by pointers along with the struct to be unpacked at the receiving side.
C2) GOP distributed encoding.
using a GOP distributed encoding style would allow slave nodes to process on different GOPs with requiring relatively no information from each other.
however, this does have some down sides:
-- a) this could potentially cause the master node to allocate a large amount of data to buffer depending on how transferring of data is done.
that is if choosing to transfer all the GOP in bulk then this would require a buffer large enough to send and recv it all. naturally this could be alleviated by transferring in smaller pieces (e.g. mini GOPs).
-- b) requires unique GOPs to exist. that is the GOP distributed encoding fails when there are not unique IDR initiated GOPs, like in the case of Periodic Intra Refresh (PIR) or Open GOP.
C3) different CPUs and related issues
as x264 works on several different CPUs you would think "oh i could have this cluster over all the different cpus it works on easily", which is sadly untrue.
x264 uses the CPU's cacheline to determine how to pad frame data in x264_frame_t's for optimal performance on that cpu.
there's a few ways you could handle this when clustering between unique cpus:
-- a) alter all the node CPUs to have the same cacheline values as the cpu within the cluster that has the largest cacheline performance value
(this would require using more memory for the cpus that had the cacheline padding increased)
-- b) alter all the node CPUs to have the smallest cacheline values of the cpu within the cluster
(this would require using less memory for the cpus that had a cacheline padding decrease at the cost of some performance)
-- c) repad the data to the receiving cpu's cacheline at send/recv ing of data
(but seriously this would be more effort than it's worth so i don't recommend it)
tl;dr:
lots of things to worry about and the finesse required to do it was a major factor in me stopping work on it.
(didn't help i was in my last semester of college while working on this and wanted to graduate more than work on it)
overall i didn't see a particular issue in using MPI-2, as the surrounding issues are not transfer protocol specific but sync specific.
MPI-2 does offer a variety of sync/async API to help out on the issue of synchronizing, so i would continue recommend using it as the implementation to do this cluster support work.
I would be willing to share the code i have, but as it is incredibly old and imo bleh now,
I would only use it now as reference code on how to do something rather than actually trying to apply it to current repository and working from there.
popper
25th January 2010, 02:02
for more input you might also look to spending some idle time on http://gogloom.com/client2/index2?mainCHAT=1&network=FreeNode&channel=%23x264 to.
i still think using a generic slax is a good way to go , incase i didnt make it clear you download a generic slax livecd, boot it, select the setup pxe server LAN booting option, and instantly you have a lan bootable slax, go over to any other PC on your LAN AND boot it into pxe mode and it will be running that exact same livecd you just booted with every module that you happen to have put on the slax livecd....
slax is the only generic liveCD/liveUSB distro i know of that comes with this pre-setup pxe setup and ready to use out the box, anywere.
OC its werth also mentioning that slax creator Tomas M is also the guy that gave you the liveCD/liveUSB scripts that everyone uses today OC
http://www.slax.org/forum.php
you could boot a native liveCD/liveUSB or just use a Vbox and liveCD/liveUSB that for background client/server LAN and Vlan use etc with all the farm and related apps in module form you require on there for on the fly mouting/loading , no messing about.
popper
25th January 2010, 04:34
"I currently work solely on a mac and would like to see it work on that but understand there are many other users in the *nix and windows world."
thats not such a big problem though,(PPC G4/5 OR X86 BTW?) all it takes is picking the right tools for your potential client/server tunnel apps to the cluster you might use....
using 'rebol view' and 'rebol core' for the GUI scripting and client/server tcp:ip udp and multicast cross platform apps today. http://re-bol.com/rebol.html http://www.rebol.com/download.html is one option to consider if your into cross platform scripting control and related apps for instance.
heres a self contained multicast rebol view script that shows you how to make and use a simple multicast whiteboard for realtime data drawing and passing ,and more.
http://www.rebolfrance.info/org/articles/multicast/sources.zip
google translate his french page if you like
http://www.rebolfrance.info/org/articles/multicast/multicast.html
you want a tiny self contained pro webserver at the heart of this cluster(s) then include The Cheyenne Web Server
http://www.rebol.com/news/cheyenne.html written in rebol, or want a cache proxy for the potential message parsing etc then
use this as your basic outline and change it as required.
http://www.rebol.org/view-script.r?script=proxy.r
perhaps you want to use java as well for part of the new farm, then theres a DHT and building a multicast protocol on top of Bamboo paper by Marcel Dischinger outlining how you might include that http://bamboo-dht.org/tutorial.html
want a basic self contained multicast ipv4 overlay tunnel for connecting remote farm sections then consider the java 'mtunnel' , alas he took the page down now but its still available...
as i said the cross platform options are many if you just consider your requirements and can figure out the best and various ways to chop up the core HD content.
how about you draw up a basic visual flow chart jpg outlining your current thinking as to all the different sections that need laying out and posting that as a start for people to see and comment on any obvious potential problems ?, nothing fancy, just a quick visual aid.
iplmedia
25th January 2010, 04:49
Thank you all for your excellent input and interest in this thread. I will consider compiling a more detail brain mapping of my thoughts for critique here. Of course I need to find the free time. At any rate, Great suggestions. Keep them coming.
Aaron
popper
25th January 2010, 05:54
OC allowing for the inclusion of a cheap freeNAS http://freenas.org/freenas or 3 :) and simple Ethernet iSCSI setup etc is always a good thing to take into account for these types of mass data storage LAN Processing Farm jobs too as its quicker than other protocols over ethernet.
http://www.microsoft.com/downloads/details.aspx?familyid=12CB3C1A-15D6-4585-B385-BEFD1319F825&displaylang=en
http://freenas.org/documentation:setup_and_user_guide:iscsi_target
http://frrl.wordpress.com/2009/08/25/fun-with-freenas-iscsi-when-a-local-disk-is-not-local/
http://www.windowsnetworking.com/articles_tutorials/Options-creating-software-based-Windows-iSCSI-Server.html
SpAwN_gUy
26th January 2010, 10:42
hi guys :) .... i'm not totally back, but anyway :)
i just thought about "how's it going on my fav x264farm?" and, viola, the thing are moving :) (i thought about "fixing" farm to be able to work with newer x264 builds. but that was just a thought :( )
i'm (as a x264farm support-unit) currently switched my job(i'm a php-dev now), so now i don't have access to a variety of HW, but i do have some ideas (if someone)... and i'll keep monitoring.. 'cause i do have alot of AVC-HD-fromMyCam-Footage to encode.. and his is a real pain in the cpu.
greets from Belarus.
popper
23rd February 2010, 13:53
well that interest didnt last very long it seems, given the TOTAL LACK of any code updates or even progress posts so far...
the intent to make something better is only as good as the coding work you actually put in and release.
Yiu
4th November 2011, 05:59
Hey Spawn Guy.
I'm thinking of getting a farm server with a number of AMD Opterons running Ubuntu/WinServer2008 and controlling it through my Win7 (x64) desktop. The problem is, I have no idea where to start when it comes to setting up the server, x264farm, and all that.
I've been using x264 for a few years now, and I've perfected a rather nifty configuration but I only get 0.45FPS on my Intel i5 750 @ 3GHz setup so I pretty much NEED the server otherwise I spend days on a single conversion.
I know I should get the system built, have a Linux/WinS version of x264 on the machine, then figure out how to set up your x264farm/MeGUI mod to work with it but I'm clueless and to be honest, scared of spending thousands on a server which may go to waste if I don't figure things out. :L
Can you give any pointers as to where to start?
SpAwN_gUy
4th November 2011, 11:17
the good thing is, that it's not so hard to set things up on a winPC.
the bad thing is... x264farm is INCOMPATIBLE with current x264 builds :(
i think the latest version supported is.. ~r900 or so..
so until someone update x264farm code to support parsing of current .stats file... i guess you'd have to stick to 1PC encoding :(
Yiu
4th November 2011, 11:25
the good thing is, that it's not so hard to set things up on a winPC.
the bad thing is... x264farm is INCOMPATIBLE with current x264 builds :(
i think the latest version supported is.. ~r900 or so..
so until someone update x264farm code to support parsing of current .stats file... i guess you'd have to stick to 1PC encoding :(
Thanks for getting back so quickly. I don't know how stupid this sounds right now but would it be possible to run x64 x264 in Win Server?
I'm probably getting the i7 3960X and OCing to ~5GHz but that's still not good enough for the amount of time it'll take.
If not,does the *nix variant of x264 support the same command-line arguments? I'm pretty sure 48 logical cores running at 2GHz will be worth getting the workhorse anyway. If possible, I'd like to raise the clock speed on the Opterons too.
48 cores X 2 GHz = 96 operations /s
V.S.
6 cores X 5 GHz X 2 (HyperThreading) = 60 operations /s
Then again, the i5 2500 beats the AMD 8 Core bulldozers so it's iffy anyway...
Sorry to hear about the compatibility problems. :(
I had been looking forward to setting up x264farm for a few months but I'm not sure if my settings file will work with the versions of x264 your application supports. If it does then it shouldn't matter, but if it's not then I'll have to build a multi-rack system instead of networking lol.
nm
4th November 2011, 12:39
Thanks for getting back so quickly. I don't know how stupid this sounds right now but would it be possible to run x64 x264 in Win Server?
Yes.
I'm probably getting the i7 3960X and OCing to ~5GHz but that's still not good enough for the amount of time it'll take.
What kind of encoding settings and filters are you tormenting yourself with?
does the *nix variant of x264 support the same command-line arguments?
Yes. But there's no native AviSynth support on other operating systems than Windows. You need Wine and avs2yuv for that.
48 cores X 2 GHz = 96 operations /s
V.S.
6 cores X 5 GHz X 2 (HyperThreading) = 60 operations /s
HyperThreading gain is x1.5 at best.
Sorry to hear about the compatibility problems. :(
I had been looking forward to setting up x264farm for a few months but I'm not sure if my settings file will work with the versions of x264 your application supports.
I'd advice against using such an old x264 version even if it supported all your settings.
Do you need to finish one encode as fast as possible, or could you simply encode separate works in parallel?
Yiu
4th November 2011, 13:01
HyperThreading gain is x1.5 at best.
Yet the 4 core i5 2500k thwarts the AMD FX 8 core processor, let alone the 2600k, 2700k, or the awaited 6 core 3960X.
What kind of encoding settings and filters are you tormenting yourself with?
A near lossless setting. No filters. Straight BluRay/H264 to x264 compressed video files. Retains approximately 99.8 per cent visual quality for animation and approximately 95 per cent visual quality for live action. It's fine-tuned for anime essentially.
Can take up to 8 days to compress a legal back-up of one of my average BluRays. I'm running out of space, so spending days getting near lossless quality AND rather small file sizes is worth it.
Do you need to finish one encode as fast as possible, or could you simply encode separate works in parallel?
Well when one whole film, lets use Transformers 2 as an example, can take up 16 days and still comes out 12GB (with DTS 5.1 in 3 languages and many other audio streams), I'd love to split it up on several machines. It's not about NEEDING speed (I.E.: Realtime), it's about not waiting for days with 100% CPU usage. If I could get encoding time for film down to 1 day (12 hours per pass, excluding audio), I'd be happy.
With this same settings file, I edited a little clip here (http://www.mediafire.com/?monpic74ip7hn9a) which is completely original (well, aside from the actual DTS Logo). It's 3.27MB and is in 1920x1080 @ 23.976FPS with FLAC 16 bit 48KHz 5.1 surround audio.
The video data for that file is only 516KB. Granted there's only 1 Keyframe and limited chroma motion, but the point is for cel and digi-cel animation it's a distribution foothold. I, by the way, am working on an animated series which is what I plan to use this for.
It's starting to look like there's no good solution.
Isn't Win Server limited to 2 physical CPUs at the moment? Or is that the previous edition?
vivan
4th November 2011, 13:37
Yet the 4 core i5 2500k thwarts the AMD FX 8 core processorAMD FX also doesn't have real 8 cores. So it's not HT win, it's AMD fail ;)
Isn't Win Server limited to 2 physical CPUs at the moment? Or is that the previous edition?http://en.wikipedia.org/wiki/Windows_Server_2008_R2#Editions - it says that it support from 1 to 64 physical CPUs depending on edition.
vBulletin® v3.8.11, Copyright ©2000-2026, vBulletin Solutions Inc.