Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > MPEG-4 AVC / H.264

Reply
 
Thread Tools Search this Thread Display Modes
Old 20th March 2007, 09:36   #201  |  Link
SpAwN_gUy
Junglist
 
SpAwN_gUy's Avatar
 
Join Date: May 2003
Location: Belarus, Minsk
Posts: 298
Quote:
Originally Posted by foxius View Post
Yeah, that's exactly the problem I'm having sometimes. One agent is stuck and others are just waiting.
let me guess.. maybe i'm having the sam sort of a problem..
it goes after al agents are ("agentname close_connection") and than controller just waits.. for something.
and all connection to source files are closed, and controller just waits..
it's not happening all the time, but some times..

this happens to me on both passes.. fist and second,.. in the end.. just before the MUXing starts ...

so my 5crnts: i think this loop occurs when agent ended encoding sended job, but something happened during sending of MKV, or Controlled didn't got it, or agent could not send it..

So a small solution,.. after a "timer" (~2 minutes) controller could ReCheck Working(as C thinks) Agents if they are actually working... timer could start after beginning of closing connections and disconnecting agents.. (well, theese are just my thoughts)
__________________
Rule Number 6: Concentrate!!!
(c)Hercules, Disney

"I like to build planes.... in the air" (c) some ADV.

tutorials
How to Setup agent-based encoding with x264farm (the easy way)

Last edited by SpAwN_gUy; 20th March 2007 at 10:03.
SpAwN_gUy is offline   Reply With Quote
Old 20th March 2007, 09:50   #202  |  Link
SpAwN_gUy
Junglist
 
SpAwN_gUy's Avatar
 
Join Date: May 2003
Location: Belarus, Minsk
Posts: 298
Quote:
Originally Posted by omion View Post
If the job is never returned, or if there is a constant cycle (the same frames get re-encoded over and over) then that's a problem.
i have now end of second-pass with 60% bitrate .. and it just reencodes something again and again..
Code:
2007-03-20~10:43:59.65  #1 7 < #2 8 (read #2)
and
2007-03-20~10:44:03.46  Number of GOPs done: 719 / 719
2007-03-20~10:44:03.46  Looks like the second pass was already done; using 0 for the number of GOPs computed
and now i've found 770 copies of GOP #658 in tempDir.. so i deleted them.. and it encodes again something.. on 100%total and 60% of Quality..

... ADD(few mins later): Yeppie.. it did.. ... two deleted GOPs(manually) and retcontrol(i guess this is the proper naming) works again.

Quote:
Originally Posted by omion View Post
I didn't find anything wrong with the MKV files you sent me. I checked them with the official mkvinfo tool and they parsed correctly. I used the same Matroska-decoder as in x264farm to merge them and everything went well... so I don't know where the problem is.
yes, mkvMerge did that also.. but not the farm
Quote:
Originally Posted by omion View Post
Are you still able to recreate it? If so, I can send you a new build which should at least give you a more descriptive error message.
yes i do,.. i thought about it,.. that you would like to recreate it again ...

... testings... so i am... it is still..
in the DOSBox
Code:
#0299: "299 - 56204 56453 5654cd.mkv"
#0300: "300 - 56454 56559 148fc1.mkv"
#0301: "301 - 56560 56617 befbc3.mkv"
Fatal error: exception Invalid_argument("Char.chr")
in the out-dump.txt
Code:
2007-03-20~11:08:53.92  Merging 88 / 302
2007-03-20~11:08:54.03  Merging 89 / 302
2007-03-20~11:08:54.17  Merging 90 / 302
2007-03-20~11:08:54.29  Merging 91 / 302
2007-03-20~11:08:54.62  Merging 92 / 302
__________________
Rule Number 6: Concentrate!!!
(c)Hercules, Disney

"I like to build planes.... in the air" (c) some ADV.

tutorials
How to Setup agent-based encoding with x264farm (the easy way)

Last edited by SpAwN_gUy; 20th March 2007 at 10:08.
SpAwN_gUy is offline   Reply With Quote
Old 20th March 2007, 19:01   #203  |  Link
omion
Registered User
 
omion's Avatar
 
Join Date: Nov 2003
Location: San Diego, CA
Posts: 325
[edit: Oops. It looks like SpAwN_gUy wrote a bunch of stuff after I started writing this. I'll get to those responses later]

First off, I'd like to mention that I updated my x264 builds to r634. links: 32-bit 64-bit

Also, I don't know if I mentioned this, but you only need my builds if you are using controller-based encoding or piped input on the agent. If you are using non-piped input to an agent-based encoder, you may use any x264 build you want.


Quote:
Originally Posted by foxius View Post
Yeah, that's exactly the problem I'm having sometimes. One agent is stuck and others are just waiting.
Does the agent show any errors? What's the CPU usage like when this happens?

One problem that I have had is that if an agent stops doing anything without informing the controller that something's wrong, then the encoding will never finish. If you restart either the controller or agent it should complete, but there's not really a good answer right now. I have been trying to think of a nice, clean way to handle this since long before I released x264farm...

Quote:
Perhaps it would work even without removing the temp files. I'll try not to delete them next time.

I'm not sure if it's possible to detect a DELFIRST dead loop, but I'd suggest putting a limit to consecutive DELFIRST messages received from an agent. If it's reached, then the controller assumes that the agent is stuch and sends a signal to drop the job and restart. Of course, this is only a quick&dirty workaround. Can't think of anything else.
Well... I really don't think DELFIRST is related to the problem you're having. The DELFIRST condition is a product of the controller so doing something with the agents won't get anywhere.

Also, if it gets into a loop, the agents will still be encoding. The loops occur when the controller throws out everything that an agent does and tells the agent to encode the same thing again. As long as --batch is set to larger than x264's --keyint option, I don't think it's possible to get in a loop.

As to what is causing the problem you're having, I don't really know. Is it doing it a lot, or was it just that one time? If multiple times, was it always the same agent that was having the problem?
__________________
"We demand rigidly defined areas of doubt and uncertainty!" - Vroomfondel, H2G2

Last edited by omion; 20th March 2007 at 19:04.
omion is offline   Reply With Quote
Old 21st March 2007, 11:36   #204  |  Link
foxius
Registered User
 
Join Date: Mar 2007
Posts: 17
Quote:
Originally Posted by omion View Post
Does the agent show any errors? What's the CPU usage like when this happens?
I don't really remember. I think there was no error displayed on the agent computer. I'll check that next time.

Quote:
Originally Posted by omion View Post
One problem that I have had is that if an agent stops doing anything without informing the controller that something's wrong, then the encoding will never finish. If you restart either the controller or agent it should complete, but there's not really a good answer right now. I have been trying to think of a nice, clean way to handle this since long before I released x264farm...
I think the controller has to "ping" the agents to get their status (I think you already do that to get the work progress). If it's stuck then you send a "restart" signal. The agents do their work in worker threads, so the main thread is always responsive.


Quote:
Originally Posted by omion View Post
Well... I really don't think DELFIRST is related to the problem you're having. The DELFIRST condition is a product of the controller so doing something with the agents won't get anywhere.
But whenever that agent is stuck in a loop I see the DELFIRST message on the controller. I guess they're related in some way.

Quote:
Originally Posted by omion View Post
As to what is causing the problem you're having, I don't really know. Is it doing it a lot, or was it just that one time? If multiple times, was it always the same agent that was having the problem?
I had it at least for each movie i recoded. For the last movie (harry potter, part 1) i had several DELFIRST & dead loops. It wasn't always the same agent.
foxius is offline   Reply With Quote
Old 28th March 2007, 14:21   #205  |  Link
foxius
Registered User
 
Join Date: Mar 2007
Posts: 17
omion,

does x264farm have any kind of IPC that lets third party applications to get some feedback from it? it'd make easier to develop a GUI or even may be integrate with MeGUI.

fox
foxius is offline   Reply With Quote
Old 29th March 2007, 08:18   #206  |  Link
SpAwN_gUy
Junglist
 
SpAwN_gUy's Avatar
 
Join Date: May 2003
Location: Belarus, Minsk
Posts: 298
Quote:
Originally Posted by foxius View Post
does x264farm have any kind of IPC that lets third party applications to get some feedback from it? it'd make easier to develop a GUI or even may be integrate with MeGUI.
i think not.. well, sources are opened..(in first post)..
as omion is not familiar with C i'm not familiar with OCaml .. i can read through code, but.. i can't programm(i've never tried in OCaml) and i don' have tools.. 'cause i'm short in traffic..

BTW... if something goes wrong.. i'm offline. ... for some time.. my boss told me that i'm not working.. and disabled iNet for me.. (eMail still works)
__________________
Rule Number 6: Concentrate!!!
(c)Hercules, Disney

"I like to build planes.... in the air" (c) some ADV.

tutorials
How to Setup agent-based encoding with x264farm (the easy way)
SpAwN_gUy is offline   Reply With Quote
Old 29th March 2007, 10:45   #207  |  Link
foxius
Registered User
 
Join Date: Mar 2007
Posts: 17
Quote:
Originally Posted by SpAwN_gUy View Post
i think not.. well, sources are opened..(in first post)..
as omion is not familiar with C i'm not familiar with OCaml .. i can read through code, but.. i can't programm(i've never tried in OCaml) and i don' have tools.. 'cause i'm short in traffic..
knowledge of C programming is not required for implementing IPC (interprocess communication).

x264farm will allow monitoring of the encoding process via IPC (named pipes, shared memory or, ideally, RPC). Then anyone could write a third party software in almost any language to communicate with x264farm.

I've googled for "ocaml RPC" and found some implementations. I'm not sure which is more suitable because I'm not familiar with the language. RPC (remote procedure calls) will allow you to define a set of functions in IDL (interface description language). IDL then can be compiled into ocaml, c++, delphi, etc. x264farn will act as an RPC server and third party software will act as a client which will be able to call functions defined in IDL.

Given that x264farm already has functions for monitoring the agents, imlementing RPC will take a few simple steps:

1. Define a set of published functions (an interface) via IDL
2. Compile IDL into ocaml
3. Map the interface to the implementation (i.e. the function defined in IDL will call the function inside x264farm)
4. Distribute the IDL file so anyone can compile it to other programming languages and use it.

RPC server and clients will be able to communicate over various protocols, such as tcp/ip, ipx, named pipes, etc.
foxius is offline   Reply With Quote
Old 5th April 2007, 19:42   #208  |  Link
Wishbringer
Silent Reader
 
Wishbringer's Avatar
 
Join Date: Dec 2003
Location: Germany
Posts: 268
Can't get an agent to run properly on a Vista32 machine.
Can start agent, but when receiving jobs I allways get a windows error message that there is something wrong with nice.exe.
Wishbringer is offline   Reply With Quote
Old 6th April 2007, 08:17   #209  |  Link
SpAwN_gUy
Junglist
 
SpAwN_gUy's Avatar
 
Join Date: May 2003
Location: Belarus, Minsk
Posts: 298
Quote:
Originally Posted by Wishbringer View Post
Can't get an agent to run properly on a Vista32 machine.
Can start agent, but when receiving jobs I allways get a windows error message that there is something wrong with nice.exe.
yeah,.. me too... i've reported that.. i have sources of newNice.exe ... and i think i'll look though a bit later, i think..
and now i'm f___ing around of HDDVD-BackUp,.. and i think AgentBased encoding will not work ...
currently i'm short on 40mins from a movie, but everythig seems fine,..
__________________
Rule Number 6: Concentrate!!!
(c)Hercules, Disney

"I like to build planes.... in the air" (c) some ADV.

tutorials
How to Setup agent-based encoding with x264farm (the easy way)
SpAwN_gUy is offline   Reply With Quote
Old 8th April 2007, 13:23   #210  |  Link
foxius
Registered User
 
Join Date: Mar 2007
Posts: 17
Quote:
Originally Posted by Wishbringer View Post
Can't get an agent to run properly on a Vista32 machine.
Can start agent, but when receiving jobs I allways get a windows error message that there is something wrong with nice.exe.
Could you please copy/paste the windows error message you're getting? Unfortunately, I don't have Vista and can't test it myself. Once I have the error message I'll try to fix it.
foxius is offline   Reply With Quote
Old 9th April 2007, 08:10   #211  |  Link
SpAwN_gUy
Junglist
 
SpAwN_gUy's Avatar
 
Join Date: May 2003
Location: Belarus, Minsk
Posts: 298
@foxius ... my old problem shows:
Code:
Леша-К7, 16.03.2007 13:24 :
Problem signature:
  Problem Event Name:	BEX
  Application Name:	nice.exe
  Application Version:	0.0.0.0
  Application Timestamp:	45f8df57
  Fault Module Name:	nice.exe
  Fault Module Version:	0.0.0.0
  Fault Module Timestamp:	45f8df57
  Exception Offset:	00005183
  Exception Code:	c000000d
  Exception Data:	00000000
  OS Version:	6.0.6000.2.0.0.256.1
  Locale ID:	1033
  Additional Information 1:	5fd9
  Additional Information 2:	de9895fb4090bedd1a27a6ff8cfedad1
  Additional Information 3:	e9b8
  Additional Information 4:	8079ce0cad7bc845eec58faba3ac0031
on Vista-Ultimate 64bit PC.
after doing this:
Code:
  nice -n 20 avs2yuv -raw -seek 1250 -frames 250 -o - "\\controllerpc\x264farm-2\CHECH-Small\nazdravicko.avs" | nice -n 20 \\controllerpc\x264farm\!AGENT\x264x64.exe --ref 16 --mixed-refs  --no-fast-pskip  --bframes 3 --b-pyramid  --b-rdo  --bime  --weightb  --direct auto --filter -2,-1 --subme 7 --trellis 2 --analyse all --8x8dct  --vbv-maxrate 25000 --me umh --threads 2 --thread-input  --progress  --no-dct-decimate  --no-psnr  --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --bitrate 292 --fps 25/1 --pass 3 --stats "temp\stats d40faa.txt"  -o "temp\output 83d1d8.mkv" - 688x400
__________________
Rule Number 6: Concentrate!!!
(c)Hercules, Disney

"I like to build planes.... in the air" (c) some ADV.

tutorials
How to Setup agent-based encoding with x264farm (the easy way)

Last edited by SpAwN_gUy; 10th April 2007 at 08:17.
SpAwN_gUy is offline   Reply With Quote
Old 10th April 2007, 16:50   #212  |  Link
Wishbringer
Silent Reader
 
Wishbringer's Avatar
 
Join Date: Dec 2003
Location: Germany
Posts: 268
@foxius

Curiously it's only a window that opens, that says that nice.exe works anymore, then another one that asks if i want to send infos to Microsoft.

But no further details what infos are available. So I can't give you a hint, sorry.

out-dump.txt from agent doesn't contain useable infos too.
Wishbringer is offline   Reply With Quote
Old 11th April 2007, 04:01   #213  |  Link
omion
Registered User
 
omion's Avatar
 
Join Date: Nov 2003
Location: San Diego, CA
Posts: 325
@all who have a problem with nice vs vista:
Does it give the error message when you try to use the nice command without x264farm? Like if you type this into the terminal:
Code:
nice -n 10 help
does it work?

@all:
Sorry about not responding in a while. I got sick again, and all the students just got back to school (I work at the UCSD bookstore), and I've been doing a major overhaul of my other program (mp3packer) so I haven't had much time for x264farm. I'll get to work on it as soon as I can, though.
__________________
"We demand rigidly defined areas of doubt and uncertainty!" - Vroomfondel, H2G2
omion is offline   Reply With Quote
Old 12th April 2007, 01:18   #214  |  Link
legoman666
Registered User
 
Join Date: Apr 2007
Posts: 142
is there a way to set the amount of frames sent to each node for each job? I have 5 computers in my network, 1 is a 1.8ghz laptop, 1 is a 1.6ghz amd xp, 1 is a p2 350mhz, 1 is an AMD x2 2.6ghz (the controller), and the last is a 1.6ghz amd 2800+.

Now of course, the p2 is a slow piece of crap. Looking at the command line output during encoding, it looks like it sends a minimum of 5250 frames for each job. After all my clients finish their work, my p2 350 is still chugging away at its job. Is there a way to set it so that each job is smaller? like 250 frames?

I somehow bugged it out so that its doing 250 frames per job, but i have no idea what I did to make it do that. any ideas?
legoman666 is offline   Reply With Quote
Old 12th April 2007, 01:52   #215  |  Link
legoman666
Registered User
 
Join Date: Apr 2007
Posts: 142
ah nevermind, i see. for the first pass its doing ~5000 frames per job, but for the second pass its a much lower number.

silly me.
legoman666 is offline   Reply With Quote
Old 12th April 2007, 20:22   #216  |  Link
Wishbringer
Silent Reader
 
Wishbringer's Avatar
 
Join Date: Dec 2003
Location: Germany
Posts: 268
Quote:
Originally Posted by omion View Post
Code:
nice -n 10 help
does it work?
@omion:
no crash, help page was displayed.

I don't really know why it crashes when starting x264.exe
Wishbringer is offline   Reply With Quote
Old 12th April 2007, 21:39   #217  |  Link
legoman666
Registered User
 
Join Date: Apr 2007
Posts: 142
this little program is amazing, thanks for the hard work!

although I do have a little question, what options do i use only for the first pass and which do i use only for the second? Or do you use them all for both. I'm using megui to get the command line instructions and just copying them for use with controller.exe. At the moment, my megui command line says (for the HQ-slowest profile):

Code:
 --pass 2 --bitrate 918 --stats ".stats" --ref 10 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-1 --subme 6 --trellis 2 --analyse all  --8x8dct --vbv-maxrate 25000 --me umh --threads auto --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --output "I:\example.mp4" "I:\example.avs"
and so for controller.exe im doing

Code:
controller.exe -B 918kbps --first "--ref 10 --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-1 --subme 6 --analyse all --merange 12 --no-psnr --trellis 2 --progress " --second "--ref 10 --no-fast-pskip --bframes 3 --b-rdo --bime --b-pyramid --weightb --direct auto --filter -2,-1 --subme 6 --trellis 2 --analyse all --8x8dct --me umh --merange 12 --no-psnr --no-dct-decimate --no-ssim --no-psnr --vbv-maxrate 25000 --progress" --avs "I:\examlpe.avs" -o C:\example.mkv
is there anything im missing/doing wrong? with these setting and 8 physical cores working on the encoding, it took ~6 hours to encode a dvd thats about 1.5 hours long, which seems reasonable considering the quality setting. Something seems to be wrong with the first pass settings, because when I used megui alone to do the encoding, the first pass was the quickest (~30fps) and the second pass was ~3.5. With my current settings for controller.exe, the first pass took ~4 hours and the 2nd ~2hours.

list of computers workin on encoding (all over 100mb/s lan)
AMD x2 3800+ @2.6ghz (controller) 2 cores (local)
AMD 1800+ (catherine)
P2 350mhz (server)
AMD A64 2800+ (basement)
Centrino 1.6ghz (mom's laptop)
2xP3 500mhz Xeon (staircase)

Speeds at which they work
Code:
Agent         | Now | GOPs | Frames | FPS  | 
Local 1       | 584 | 191  | 35515  | 9.11 | 
Local 2       | 395 | 176  | 35362  | 9.61 | 
basemnet 1   | 610 | 139  | 24938  | 6.18 | 
server 1      | 644 | 7    | 1448   | 0.33 | 
mom's laptop 1 | x   | 40   | 7095   | 3.87 | 
catherine 1  | x   | 106  | 18011  | 4.59 | 
staircase 1   | 659 | 32   | 6050   | 1.43 | 
staircase 2   | 583 | 26   | 5110   | 1.13 |

Last edited by legoman666; 12th April 2007 at 21:46.
legoman666 is offline   Reply With Quote
Old 13th April 2007, 08:48   #218  |  Link
Wishbringer
Silent Reader
 
Wishbringer's Avatar
 
Join Date: Dec 2003
Location: Germany
Posts: 268
@legoman666

could it be that megui uses lesser quality settings for the first pass?
normally you don't need all options in first pass (like -b-rdo).
best is to look at megui's log to see, what commandline it used for first pass.
Wishbringer is offline   Reply With Quote
Old 13th April 2007, 13:47   #219  |  Link
legoman666
Registered User
 
Join Date: Apr 2007
Posts: 142
ah thanks for the suggestion, that helps

Code:
controller.exe -B 8079kbps --first "--bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --progress --no-dct-decimate --no-psnr --no-ssim" --second "--ref 10 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-1 --subme 6 --trellis 2 --analyse all  --8x8dct --vbv-maxrate 25000 --me umh --progress --no-dct-decimate --no-psnr --no-ssim" --avs "I:\movie.avs" -o C:\movie.mkv
theres my next project, 720p uber quality . should take.... 12-14 hours for my 6 computers.
legoman666 is offline   Reply With Quote
Old 13th April 2007, 14:15   #220  |  Link
SpAwN_gUy
Junglist
 
SpAwN_gUy's Avatar
 
Join Date: May 2003
Location: Belarus, Minsk
Posts: 298
Quote:
Originally Posted by legoman666 View Post
although I do have a little question, what options do i use only for the first pass and which do i use only for the second? Or do you use them all for both. I'm using megui to get the command line instructions and just copying them for use with controller.exe. At the moment, my megui command line says (for the HQ-slowest profile):
okay.. i'm back,.. and i don't have iNet.. ...
so,.. i have few suggestions for you ..
1) i've noticed that most powerfull PCs sould be first in controller's config (i have something like 10PCs Farm ade powers vary alot)
2) .. well.. if you add Job in MeGUI and then close it.. you'll get 2 job.xml files .. look through them and you'll find something like <Commandline> .... copy the values from both files and you'll see the difference..
2b) if someone wants i can send a beta-controllerGUI prog (and agentGUI) ... main thing is that controllerGUI can generate commandline for the controller (but don't even try to start it for now ... it'll give you a headake)..

peace to all, i'm waiting for something new ...
__________________
Rule Number 6: Concentrate!!!
(c)Hercules, Disney

"I like to build planes.... in the air" (c) some ADV.

tutorials
How to Setup agent-based encoding with x264farm (the easy way)
SpAwN_gUy is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 14:21.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2017, vBulletin Solutions Inc.