Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
25th February 2007, 09:44 | #181 | Link |
Registered User
Join Date: Feb 2005
Posts: 16
|
hrm, I can get this working on local worker but the 2nd machine I have doesn't appear to want to encode anything. i'm still plugging away at it. the network paths and setup was a tad difficult, but I think i've got that all worked out.
for some reason right now, the remote agent i have, receives a command to encode. yet it doesn't appear to be encoding. I did have a question. regarding agent based encoding. If you are doing this, can the input video in the avs be on a network share for remote agents? Both my agents are using the same .avs and I just have the AVS looking for the input video on a Network drive I mapped(which is actually a local share on the controll/local agent machine). I guess thats not really a good question, I guess since they both use the same avs, they both have to have the same access to the input video and the path has to be the same, otherwise it won't work. Anyhow, not sure whats going on with this remote agent, I can't get it to actually encode anything for nothing. The second pass will be up in an hour or 2, perhaps it will have better luck on that pass. :s *edit* i think i figured it out, i guess even tho i had the base (on remote agent) pointing at the network share, ineeded the avs actually in the agent directory instead. appears to be encoding fine now. Last edited by theturtle; 25th February 2007 at 13:32. |
25th February 2007, 15:14 | #182 | Link |
Registered User
Join Date: Feb 2005
Posts: 16
|
i thought i'd post this separately, i just finished an encode with 2 computers local agent and remote agent. the resulting encode has flashes of grey every few seconds or so. I thought at first this was during scene changes but its not. Its like a blank spot in the video or something. Perhaps made during merging process? :s
|
25th February 2007, 18:55 | #183 | Link | |
Registered User
Join Date: Nov 2003
Location: San Diego, CA
Posts: 325
|
Quote:
Also, you can find out where the gray occurs by doing the following: Turn off all the agents except for one (in the config file, or just don't start the other agents up) Run the controller with the --keeptemp option (and probably also --frames xyz and --restart to make it fast and repeatable) If the resultant file does not have the gray frames, then the problem is with another agent If the resultant file does have the gray frames, then you can check the temp directory and see if the raw pieces have gray frames. If they don't then it's probably the merger that's doing it. Also, just to make sure, the initial AVS file doesn't have the gray frames, right? @SpAwN_gUy: Sorry about the delay in fixing nice. I don't really have a lot of time lately, but I should get it done by Wednesday this week
__________________
"We demand rigidly defined areas of doubt and uncertainty!" - Vroomfondel, H2G2 |
|
25th February 2007, 20:54 | #184 | Link | |
Registered User
Join Date: Mar 2005
Location: Germany
Posts: 27
|
Quote:
Try to play the script with the WMP. The grey frames should also appear here! Did you try to reinstall your avisynth or tried a different version? That's what helped me getting rid of this bug... greetings to all x264farmers %-) |
|
26th February 2007, 02:24 | #185 | Link | ||
Registered User
Join Date: Feb 2005
Posts: 16
|
Quote:
*update* I just installed both of these and it appears its encoding, the network utilization skyrocketed from my encode last night. it was hopping around 20-50%, now its about 99% steady. perhaps that was my problem. hopefully it doesn't produce the same gray frames. we'll see. mooore testing Quote:
Last edited by theturtle; 26th February 2007 at 05:14. |
||
26th February 2007, 14:10 | #187 | Link | |
Registered User
Join Date: Feb 2005
Posts: 16
|
Quote:
http://forum.doom9.org/showthread.ph...t=x264farm+gui dunno how well it works, the cli version took a bit of figuring and alot of mistakes on my part, but wasn't too difficult. |
|
5th March 2007, 09:54 | #188 | Link | |
Junglist
Join Date: May 2003
Location: Belarus, Minsk
Posts: 298
|
Quote:
agentGUI has an UnKILLable option (if agent crashes it is auto-Restarted) and can minimize to tray. controllerGUI has "Import MeGUI Jobs" button, and actually imports them,.. and generates command lide for starting controller.. ... that is for now,.. lots of ideas, and no time at all.. @ALL is the Tutorial i wrote usefull? or maybe...??
__________________
Rule Number 6: Concentrate!!! (c)Hercules, Disney "I like to build planes.... in the air" (c) some ADV. tutorials How to Setup agent-based encoding with x264farm (the easy way) Last edited by SpAwN_gUy; 5th March 2007 at 09:57. |
|
11th March 2007, 08:32 | #189 | Link |
Registered User
Join Date: Nov 2003
Location: San Diego, CA
Posts: 325
|
1.09 is out (or will be shortly, when my server reboots...)
It should be more accurate with the bitrate target, and the agents now use nice for avs2yuv (if both niceness and piped input are used) I didn't add bitrate in the same-file detection (because I only remembered it just now) Also, the agents will by default delete any temp file which is older than 1 week. I noticed my temp directory filling up with a bunch of junk after testing x264farm, and I decided to do something about it. By default, a cleanup thread will wake up once a day and delete all the temp files that are over 1 week old. The amount of time is adjustable with the --stale <x> command-line option. <x> is the maximum age that a temp file can be, in seconds. The release took so long because I've been trying to rewrite the nice.exe program. I think 1.1MB is way too big for a program like that, and the fact that it can't run network things is not so good either. However. in order to easily access process priorities I have had to use C/C++ (which I don't really know) and the whole thing ended up being extraordinarily frustrating and a huge waste of time. So now, instead of a new, slimmed-down nice.exe program, I have a reminder of why I program with OCaml in the first place
__________________
"We demand rigidly defined areas of doubt and uncertainty!" - Vroomfondel, H2G2 |
12th March 2007, 14:36 | #190 | Link |
Junglist
Join Date: May 2003
Location: Belarus, Minsk
Posts: 298
|
Okay, done some testings...
junk removal on agent PCs - works(i think.. it shows, that some filas are too old..) but,.. niceness of avs2yuv + network x264 - doesn't ... well.. it's not working now at all.. agentpipe=1 Code:
Testing file "\\PC201\x264farm-2\xFiles\S1p2\S1p2.avs": FOUND! 2 agent-based encoding 2 doing the following: nice -n 20 avs2yuv -raw -seek 0 -frames 5223 -o - "\\PC201\x264farm-2\xFiles\S1p2\S1p2.avs" | nice -n 20 \\PC201\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\stats 41425d.txt" -o NUL - 576x432 Avisynth error: Import: couldn't open "D:\PC201\x264farm-2\xFiles\S1p2\S1p2.avs" Code:
Testing file "\\PC201\x264farm-2\xFiles\S1p2\S1p2.avs": FOUND! 7 agent-based encoding 7 doing the following: nice -n 20 \\PC201\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid --directauto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\stats e31f8a.txt" --seek 0 --frames 5223 -o NUL "\\PC201\x264farm-2\xFiles\S1p2\S1p2.avs" avis [error]: unsupported input format (DIB ) x264 [error]: could not open input file '\PC201\x264farm-2\xFiles\S1p2\S1p2.avs' 7 exited 255 Thread received exception "Failure(\"ERROR: nonzero return code from x264 (255)\")" i guess.. i'm going back to controller-1.09 and agent-1.08... waiting for nice, only ... maybe.. it would be better to change nice on windows machines to something else? ... or then we will loose crossPlatform functionality? yeah,.. and i've noticed... for faster functionality(encoding) in controller's config-file place most powerfull stations first. ... Testing (c1.09+a1.08) in progress... i'm currently looking on first-pass FPSes.. it's .. em.. 138FPS... AWESOME!!!!!!!! and i think it's rising at the end it was 158... i haven't seen hang on the end of first pass on controller.. cool ... 37fps on start of second pass.. and rising ... i think. today is a good day,.. i'm happy really ..
__________________
Rule Number 6: Concentrate!!! (c)Hercules, Disney "I like to build planes.... in the air" (c) some ADV. tutorials How to Setup agent-based encoding with x264farm (the easy way) Last edited by SpAwN_gUy; 12th March 2007 at 15:26. |
12th March 2007, 19:54 | #191 | Link | ||||
Registered User
Join Date: Nov 2003
Location: San Diego, CA
Posts: 325
|
Quote:
I'll see what I can do about that program. And just in case you were wondering: Quote:
priorityprog.exe LOW runthis a b c dit behaves exactly like typing in runthis a b c dwith a LOW priority. I have so far not found a solution for this. Quote:
With a lot of computers (like your setup) I can believe that that makes a big difference. Quote:
__________________
"We demand rigidly defined areas of doubt and uncertainty!" - Vroomfondel, H2G2 |
||||
13th March 2007, 12:33 | #192 | Link | ||
Junglist
Join Date: May 2003
Location: Belarus, Minsk
Posts: 298
|
YEPPIE.... i've figured all out ... i just remembered, when i was trying to run agent as service, there was an advice..(at the end of the post,.. add some mystery )
Quote:
it is ConsoleApp and uses WinAPI (CreateProcess with Priority flag set).. and i've found in old winSDK only four priorities.. IDLE, NORMAL, HIGH and REALTIME... and i dunno wich are usefull .. anyway results -> later. --and-now-------------------------- Quote:
Code:
nice -n 20 \\PC201\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\stats e31f8a.txt" --seek 0 --frames 5223 -o NUL "//PC201/x264farm-2/xFiles/S1p2/S1p2.avs" nice -n 20 //PC201/x264farm/!AGENT/x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\stats e31f8a.txt" --seek 0 --frames 5223 -o NUL "//PC201/x264farm-2/xFiles/S1p2/S1p2.avs" Code:
nice -n 20 //PC201/x264farm/!AGENT/x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\stats e31f8a.txt" --seek 0 --frames 5223 -o NUL "\\PC201\x264farm-2\xFiles\S1p2\S1p2.avs" and then after: Code:
nice -n 20 avs2yuv -raw -seek 224022 -frames 4192 -o - "//PC201/x264farm-2/xFiles/S1p2/S1p2.avs" | nice -n 20 \\PC201\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\224022 228213 188ff7.txt" -o NUL - 576x432 nice -n 20 avs2yuv -raw -seek 224022 -frames 4192 -o - "//PC201/x264farm-2/xFiles/S1p2/S1p2.avs" | nice -n 20 //PC201/x264farm/!AGENT/x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\224022 228213 188ff7.txt" -o NUL - 576x432 Code:
nice -n 20 avs2yuv -raw -seek 224022 -frames 4192 -o - "\\PC201\x264farm-2\xFiles\S1p2\S1p2.avs" | nice -n 20 \\PC201\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\224022 228213 188ff7.txt" -o NUL - 576x432 maybe.. it is the best way.. to change those slashes back?
__________________
Rule Number 6: Concentrate!!! (c)Hercules, Disney "I like to build planes.... in the air" (c) some ADV. tutorials How to Setup agent-based encoding with x264farm (the easy way) |
||
14th March 2007, 00:17 | #193 | Link |
Registered User
Join Date: Nov 2003
Location: San Diego, CA
Posts: 325
|
@SpAwN_gUy:
So it looks like everything works when using forward slashes rather than backslashes? That's good to know. It's looking like it won't matter for much longer, though. I was just contacted by foxius (at least, that will be his name when his 5 days are up ) who is making a nice alternative to the nice.exe program I am currently distributing. It will be a drop-in replacement, and should weigh in at ~50KB. I can't say for sure, but it looks like it should be done soon (depending on how many more unreasonable requests I make )
__________________
"We demand rigidly defined areas of doubt and uncertainty!" - Vroomfondel, H2G2 |
15th March 2007, 07:29 | #194 | Link |
Registered User
Join Date: Nov 2003
Location: San Diego, CA
Posts: 325
|
All right! I finally updated nice.exe!
(or rather, foxius updated it and I took credit for it ) Get it here (source). I also updated the first post's link. Awesome new features:
MAJOR props to foxius! Thanks a bunch!
__________________
"We demand rigidly defined areas of doubt and uncertainty!" - Vroomfondel, H2G2 |
15th March 2007, 10:35 | #195 | Link |
Junglist
Join Date: May 2003
Location: Belarus, Minsk
Posts: 298
|
) .... updated to newNice ... and looked through source..
well.. it seems, like it is doing the same thing i was thinkikng of.. and made actually ... Code:
winAPI: CreateProcess(with parameters) currently i'm doing... oops.. done HDTV-Encode.. oops.. errore again. (something strange on merging GOPs.. c1.08 andc1.09 are doing all the same..). deleted those GOPs.. redoing them.. the second pass fps ~5-6FPS.. i think this value is.. like.. very cool for HDTV-Encode ... and again... something strange.. in log last line is "Merging 92 / 302". Deleted 3 gops (92, 93, 94).. and waiting. btw. feature request ... can you delete unNeeded GOPs during the second-pass? (i was looking through controller's source and found a line like "it's not an easy job to find GOPs" .. or something). so i thought.. if UnNedded(ReEncoded GOPs) are deleted during second-pass, so after crash you wouldn't need to calculate something.. (or the whole thing is working different?) and one more thing.. about "more accurate bitrate". with no additional params(overrides).. two of my encodes were +100Mb.. instead of 1460 they are 1560.. i guess i'm back to "--3thresh 0.95 --3ratio 0.2" okay,.. i've done "pause" in .bat file... so.. Code:
Fatal error: exception Invalid_argument("Char.chr") and the file is allways like 211Mbs...
__________________
Rule Number 6: Concentrate!!! (c)Hercules, Disney "I like to build planes.... in the air" (c) some ADV. tutorials How to Setup agent-based encoding with x264farm (the easy way) Last edited by SpAwN_gUy; 15th March 2007 at 11:41. |
16th March 2007, 07:41 | #196 | Link | |||||
Registered User
Join Date: Nov 2003
Location: San Diego, CA
Posts: 325
|
Quote:
And about me using foxius's code rather than yours, well... he contacted me first Don't worry; I'll tell you the next time I need a program that I don't want to write Quote:
Quote:
That comment about finding GOPs (around line 450 in pass2.ml) was actually referring to the fact that there could be many more GOPs done than are in the temp stats file, and how it is quite difficult to determine where to restart counting when the program is re-run. However, the only thing deleting the unneeded GOPs will get you is extra disk space. Everything is calculated based on the working_stats#.txt files. The result is that, once the reference to the old GOP is overwritten by the new GOP file name, the old GOP file might as well not exist any more. Nothing in the program references old GOP files, so they don't have any effect on anything. This means that there is no reason not to delete them, but it won't help the program if they are deleted. In either case, the whole temp directory is cleaned up at the end of the second pass, so the final result is the same no matter what files are in there. Quote:
I probably have a plus when I should have a minus or something. Due to the self-correcting nature of ratecontrol, just about every bug in it will result in something that is a bit off, but not by much. Just curious: what are you using to measure bitrate? The bitrate that x264farm (and x264) use are raw AVC bits. The container always adds overhead to that. It shouldn't be 8%, though, so I think something's wrong anyway. Quote:
(apologies for any horrible grammar or anything in this post. My brain is not working properly right now. Too much caffeine... need more sleep...)
__________________
"We demand rigidly defined areas of doubt and uncertainty!" - Vroomfondel, H2G2 |
|||||
16th March 2007, 09:11 | #197 | Link | ||||
Junglist
Join Date: May 2003
Location: Belarus, Minsk
Posts: 298
|
Quote:
Quote:
and, i've muxed with my bare hands (mkvmerged) all those .mkv-files.. some of them were "unKnown format" (i guess one of those zero-bytes file, i have in _temp), and (i guess i forgot some of files, but.. ~godd mkv.. (it became 1 second less)) the end file is good,.. (thise 91,92,93 - were good in the end). so i'm sending you those files.. via e-mail.. Quote:
Quote:
btw. minasan, how would you like to start controller? invisible and all "usefull info is displayed in GUI"(as meGUI does) or just run a pregenerated comLine, while showing DOSBox(like GKnot does)? yeah,.. and something like a BUG,.. i think.. after "resuming" on second-pass,.. the info(that periodically is shown..) does not show "Current GOP" (column "Now") just "x"..
__________________
Rule Number 6: Concentrate!!! (c)Hercules, Disney "I like to build planes.... in the air" (c) some ADV. tutorials How to Setup agent-based encoding with x264farm (the easy way) Last edited by SpAwN_gUy; 16th March 2007 at 09:20. |
||||
19th March 2007, 09:22 | #198 | Link |
Registered User
Join Date: Mar 2007
Posts: 17
|
Finally, I have a voice =)
x264farm is working like a charm here, the only bug I've found so far is that sometimes I get a DELFIRST from an agent and it doesn't resolve until I kill the agent, clean his temp folder and restart. I think, it would be more convenient if the agent could clean the broken files itself. I've been using x264farm for a week now and it saved me lots of hours. I have 7 agents: 1. 1x working on the controller, Pentium D 820 2.8GHz, gives ~7-8fps in the second pass 2. 3x on the same 100mbit switch: - Celeron D 2.6GHz, ~3-4fps - Celeron D 2.6GHz, ~3-4fps - Pentium 3.2GHz with HT, ~4-5fps 3. 3x on another switch over 100mbit uplink: - Pentium 2.8GHz with HT, ~3fps - Pentium 2.8GHz with HT, ~3fps - Pentium 2.8GHz, ~3fps Using agent based encoding gave me like 4 times speed up. Obviously, I'm losing 1-2fps because of the slow uplink. Sending 2 jobs to a CPU with HT increases the performance only by 0.3-0.4fps while putting more load on the network. |
19th March 2007, 19:09 | #199 | Link | |
Registered User
Join Date: Nov 2003
Location: San Diego, CA
Posts: 325
|
Quote:
It turns out that dealing with this case correctly is quite difficult, so I added it to the output so I could properly debug it. If the job is never returned, or if there is a constant cycle (the same frames get re-encoded over and over) then that's a problem. It is, however, extremely odd that you need to clean out the agent's temp folder manually. The agent has no memory of any previous jobs when it gets restarted. When a new request is made, the agent picks completely new temp files for the encoding. It wouldn't even know where to check for the old temp files, as all the file names are generated randomly. @SpAwN_gUy: I didn't find anything wrong with the MKV files you sent me. I checked them with the official mkvinfo tool and they parsed correctly. I used the same Matroska-decoder as in x264farm to merge them and everything went well... so I don't know where the problem is. Are you still able to recreate it? If so, I can send you a new build which should at least give you a more descriptive error message. And sorry about the delay. I thought I would get all this stuff done last week, but I got sick. And I play too much Oblivion. (Maybe I got sick from playing Oblivion...?)
__________________
"We demand rigidly defined areas of doubt and uncertainty!" - Vroomfondel, H2G2 |
|
19th March 2007, 20:06 | #200 | Link | ||
Registered User
Join Date: Mar 2007
Posts: 17
|
Quote:
Quote:
I'm not sure if it's possible to detect a DELFIRST dead loop, but I'd suggest putting a limit to consecutive DELFIRST messages received from an agent. If it's reached, then the controller assumes that the agent is stuch and sends a signal to drop the job and restart. Of course, this is only a quick&dirty workaround. Can't think of anything else. |
||
Thread Tools | Search this Thread |
Display Modes | |
|
|