Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > MPEG-4 AVC / H.264

Reply
 
Thread Tools Search this Thread Display Modes
Old 25th February 2007, 09:44   #181  |  Link
theturtle
Registered User
 
Join Date: Feb 2005
Posts: 16
hrm, I can get this working on local worker but the 2nd machine I have doesn't appear to want to encode anything. i'm still plugging away at it. the network paths and setup was a tad difficult, but I think i've got that all worked out.

for some reason right now, the remote agent i have, receives a command to encode. yet it doesn't appear to be encoding.

I did have a question. regarding agent based encoding. If you are doing this, can the input video in the avs be on a network share for remote agents? Both my agents are using the same .avs and I just have the AVS looking for the input video on a Network drive I mapped(which is actually a local share on the controll/local agent machine). I guess thats not really a good question, I guess since they both use the same avs, they both have to have the same access to the input video and the path has to be the same, otherwise it won't work.

Anyhow, not sure whats going on with this remote agent, I can't get it to actually encode anything for nothing. The second pass will be up in an hour or 2, perhaps it will have better luck on that pass. :s

*edit* i think i figured it out, i guess even tho i had the base (on remote agent) pointing at the network share, ineeded the avs actually in the agent directory instead. appears to be encoding fine now.

Last edited by theturtle; 25th February 2007 at 13:32.
theturtle is offline   Reply With Quote
Old 25th February 2007, 15:14   #182  |  Link
theturtle
Registered User
 
Join Date: Feb 2005
Posts: 16
i thought i'd post this separately, i just finished an encode with 2 computers local agent and remote agent. the resulting encode has flashes of grey every few seconds or so. I thought at first this was during scene changes but its not. Its like a blank spot in the video or something. Perhaps made during merging process? :s
theturtle is offline   Reply With Quote
Old 25th February 2007, 18:55   #183  |  Link
omion
Registered User
 
omion's Avatar
 
Join Date: Nov 2003
Location: San Diego, CA
Posts: 325
Quote:
Originally Posted by theturtle View Post
i thought i'd post this separately, i just finished an encode with 2 computers local agent and remote agent. the resulting encode has flashes of grey every few seconds or so. I thought at first this was during scene changes but its not. Its like a blank spot in the video or something. Perhaps made during merging process? :s
Are there entire frames which are gray, or does it seem to start in the middle of one frame and end in the middle of another?

Also, you can find out where the gray occurs by doing the following:
Turn off all the agents except for one (in the config file, or just don't start the other agents up)
Run the controller with the --keeptemp option (and probably also --frames xyz and --restart to make it fast and repeatable)
If the resultant file does not have the gray frames, then the problem is with another agent
If the resultant file does have the gray frames, then you can check the temp directory and see if the raw pieces have gray frames. If they don't then it's probably the merger that's doing it.

Also, just to make sure, the initial AVS file doesn't have the gray frames, right?


@SpAwN_gUy:
Sorry about the delay in fixing nice. I don't really have a lot of time lately, but I should get it done by Wednesday this week
__________________
"We demand rigidly defined areas of doubt and uncertainty!" - Vroomfondel, H2G2
omion is offline   Reply With Quote
Old 25th February 2007, 20:54   #184  |  Link
AVmaniac
Registered User
 
AVmaniac's Avatar
 
Join Date: Mar 2005
Location: Germany
Posts: 27
Quote:
Originally Posted by theturtle View Post
i thought i'd post this separately, i just finished an encode with 2 computers local agent and remote agent. the resulting encode has flashes of grey every few seconds or so. I thought at first this was during scene changes but its not. Its like a blank spot in the video or something. Perhaps made during merging process? :s
I don't think it's a bug of the merging part.
Try to play the script with the WMP.
The grey frames should also appear here!
Did you try to reinstall your avisynth or tried a different version?
That's what helped me getting rid of this bug...

greetings to all x264farmers %-)
AVmaniac is offline   Reply With Quote
Old 26th February 2007, 02:24   #185  |  Link
theturtle
Registered User
 
Join Date: Feb 2005
Posts: 16
Quote:
Originally Posted by theturtle View Post
*edit* i think i figured it out, i guess even tho i had the base (on remote agent) pointing at the network share, ineeded the avs actually in the agent directory instead. appears to be encoding fine now.
^^^^ regarding this. For some reason if I set the <base> to my network drive that I mapped, my remote agent executes x264.exe but yet cpu is at 0%. If I copy the avs to the remote agent's agent-win32 directory, it appears to encode fine. Although I've just noticed, my source is h.264 mkv. I didn't have either ffdshow or haali media splitter installed on my agent. Should I need these? I figured the input to the agent was yuv from the controller.


*update* I just installed both of these and it appears its encoding, the network utilization skyrocketed from my encode last night. it was hopping around 20-50%, now its about 99% steady. perhaps that was my problem. hopefully it doesn't produce the same gray frames. we'll see. mooore testing

Quote:
Originally Posted by AVmaniac View Post
I don't think it's a bug of the merging part.
Try to play the script with the WMP.
The grey frames should also appear here!
Did you try to reinstall your avisynth or tried a different version?
That's what helped me getting rid of this bug...

greetings to all x264farmers %-)
just talked with someone about this, they seem to be pretty confident its an issue with directshowsource. my source is h.264, i'm going to try the new avc/h.264 avs source plugin. I was getting blank gray frames in the source with directshowsource btw, so its definitely not a x264farm issue.

Last edited by theturtle; 26th February 2007 at 05:14.
theturtle is offline   Reply With Quote
Old 26th February 2007, 12:08   #186  |  Link
mpioner
Registered User
 
Join Date: Sep 2005
Posts: 15
omion
you thought about GUI?
Maybe anybody will write path for meGUI for support x264farm.
mpioner is offline   Reply With Quote
Old 26th February 2007, 14:10   #187  |  Link
theturtle
Registered User
 
Join Date: Feb 2005
Posts: 16
Quote:
Originally Posted by mpioner View Post
omion
you thought about GUI?
Maybe anybody will write path for meGUI for support x264farm.
tried this?
http://forum.doom9.org/showthread.ph...t=x264farm+gui

dunno how well it works, the cli version took a bit of figuring and alot of mistakes on my part, but wasn't too difficult.
theturtle is offline   Reply With Quote
Old 5th March 2007, 09:54   #188  |  Link
SpAwN_gUy
Junglist
 
SpAwN_gUy's Avatar
 
Join Date: May 2003
Location: Belarus, Minsk
Posts: 298
Quote:
Originally Posted by mpioner View Post
omion
you thought about GUI?
Maybe anybody will write path for meGUI for support x264farm.
i'm playing around with Delphi... for like some months.. and it seems like a testing version will be shown in few weeks.. i guess..(well it's all unofficial)..

agentGUI has an UnKILLable option (if agent crashes it is auto-Restarted) and can minimize to tray.
controllerGUI has "Import MeGUI Jobs" button, and actually imports them,.. and generates command lide for starting controller.. ... that is for now,.. lots of ideas, and no time at all..

@ALL
is the Tutorial i wrote usefull? or maybe...??
__________________
Rule Number 6: Concentrate!!!
(c)Hercules, Disney

"I like to build planes.... in the air" (c) some ADV.

tutorials
How to Setup agent-based encoding with x264farm (the easy way)

Last edited by SpAwN_gUy; 5th March 2007 at 09:57.
SpAwN_gUy is offline   Reply With Quote
Old 11th March 2007, 08:32   #189  |  Link
omion
Registered User
 
omion's Avatar
 
Join Date: Nov 2003
Location: San Diego, CA
Posts: 325
1.09 is out (or will be shortly, when my server reboots...)

It should be more accurate with the bitrate target, and the agents now use nice for avs2yuv (if both niceness and piped input are used)
I didn't add bitrate in the same-file detection (because I only remembered it just now)

Also, the agents will by default delete any temp file which is older than 1 week. I noticed my temp directory filling up with a bunch of junk after testing x264farm, and I decided to do something about it.
By default, a cleanup thread will wake up once a day and delete all the temp files that are over 1 week old. The amount of time is adjustable with the --stale <x> command-line option. <x> is the maximum age that a temp file can be, in seconds.


The release took so long because I've been trying to rewrite the nice.exe program. I think 1.1MB is way too big for a program like that, and the fact that it can't run network things is not so good either. However. in order to easily access process priorities I have had to use C/C++ (which I don't really know) and the whole thing ended up being extraordinarily frustrating and a huge waste of time. So now, instead of a new, slimmed-down nice.exe program, I have a reminder of why I program with OCaml in the first place
__________________
"We demand rigidly defined areas of doubt and uncertainty!" - Vroomfondel, H2G2
omion is offline   Reply With Quote
Old 12th March 2007, 14:36   #190  |  Link
SpAwN_gUy
Junglist
 
SpAwN_gUy's Avatar
 
Join Date: May 2003
Location: Belarus, Minsk
Posts: 298
Okay, done some testings...

junk removal on agent PCs - works(i think.. it shows, that some filas are too old..)
but,.. niceness of avs2yuv + network x264 - doesn't ... well.. it's not working now at all..

agentpipe=1
Code:
Testing file "\\PC201\x264farm-2\xFiles\S1p2\S1p2.avs": FOUND!
2 agent-based encoding
2 doing the following:
  nice -n 20 avs2yuv -raw -seek 0 -frames 5223 -o - "\\PC201\x264farm-2\xFiles\S1p2\S1p2.avs" | nice -n 20 \\PC201\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid  --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input  --progress  --no-dct-decimate  --no-psnr  --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\stats 41425d.txt"  -o NUL - 576x432
Avisynth error:
Import: couldn't open "D:\PC201\x264farm-2\xFiles\S1p2\S1p2.avs"
agentpipe=0
Code:
Testing file "\\PC201\x264farm-2\xFiles\S1p2\S1p2.avs": FOUND!
7 agent-based encoding
7 doing the following:
  nice -n 20 \\PC201\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid  --directauto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input  --progress  --no-dct-decimate  --no-psnr  --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\stats e31f8a.txt"  --seek 0 --frames 5223 -o NUL "\\PC201\x264farm-2\xFiles\S1p2\S1p2.avs"
avis [error]: unsupported input format (DIB )
x264 [error]: could not open input file '\PC201\x264farm-2\xFiles\S1p2\S1p2.avs'

7 exited 255
Thread received exception "Failure(\"ERROR: nonzero return code from x264 (255)\")"
i've downloaded archive woth nice once again,.. and i think it have not being changed..

i guess.. i'm going back to controller-1.09 and agent-1.08... waiting for nice, only ...

maybe.. it would be better to change nice on windows machines to something else? ... or then we will loose crossPlatform functionality?

yeah,.. and i've noticed... for faster functionality(encoding) in controller's config-file place most powerfull stations first. ...

Testing (c1.09+a1.08) in progress...
i'm currently looking on first-pass FPSes.. it's .. em.. 138FPS... AWESOME!!!!!!!! and i think it's rising
at the end it was 158...

i haven't seen hang on the end of first pass on controller.. cool ...
37fps on start of second pass.. and rising ...

i think. today is a good day,.. i'm happy really ..
__________________
Rule Number 6: Concentrate!!!
(c)Hercules, Disney

"I like to build planes.... in the air" (c) some ADV.

tutorials
How to Setup agent-based encoding with x264farm (the easy way)

Last edited by SpAwN_gUy; 12th March 2007 at 15:26.
SpAwN_gUy is offline   Reply With Quote
Old 12th March 2007, 19:54   #191  |  Link
omion
Registered User
 
omion's Avatar
 
Join Date: Nov 2003
Location: San Diego, CA
Posts: 325
Quote:
Originally Posted by SpAwN_gUy View Post
Okay, done some testings...

junk removal on agent PCs - works(i think.. it shows, that some filas are too old..)
but,.. niceness of avs2yuv + network x264 - doesn't ... well.. it's not working now at all..
Grrr... Nice doesn't seem to like two backslashes in a row. Nice is not very nice. I'm really not too happy with that program, which is why I've been trying to write my own, only to learn that I'm very unhappy with C.

I'll see what I can do about that program.

And just in case you were wondering:

Quote:
maybe.. it would be better to change nice on windows machines to something else? ... or then we will loose crossPlatform functionality?
I could make Windows-compiles do different things from the other platforms. The "start" program does basically the same thing, except it's harder to stop it once it started. The problem with all of these programs is that they have side-effects. I really need a program that when you type in
priorityprog.exe LOW runthis a b c d
it behaves exactly like typing in
runthis a b c d
with a LOW priority.
I have so far not found a solution for this.

Quote:
yeah,.. and i've noticed... for faster functionality(encoding) in controller's config-file place most powerfull stations first. ...
Once everything starts up there isn't a difference, but the topmost agents will get the first splits, resulting in a faster startup.

With a lot of computers (like your setup) I can believe that that makes a big difference.

Quote:
Testing (c1.09+a1.08) in progress...
i'm currently looking on first-pass FPSes.. it's .. em.. 138FPS... AWESOME!!!!!!!! and i think it's rising
at the end it was 158...

i haven't seen hang on the end of first pass on controller.. cool ...
37fps on start of second pass.. and rising ...

i think. today is a good day,.. i'm happy really ..
Good to hear! Thanks!
__________________
"We demand rigidly defined areas of doubt and uncertainty!" - Vroomfondel, H2G2
omion is offline   Reply With Quote
Old 13th March 2007, 12:33   #192  |  Link
SpAwN_gUy
Junglist
 
SpAwN_gUy's Avatar
 
Join Date: May 2003
Location: Belarus, Minsk
Posts: 298
YEPPIE.... i've figured all out ... i just remembered, when i was trying to run agent as service, there was an advice..(at the end of the post,.. add some mystery )
Quote:
Originally Posted by omion View Post
And just in case you were wondering:
I could make Windows-compiles do different things from the other platforms. The "start" program does basically the same thing, except it's harder to stop it once it started. The problem with all of these programs is that they have side-effects. I really need a program that when you type in
priorityprog.exe LOW runthis a b c d
it behaves exactly like typing in
runthis a b c d
with a LOW priority.
I have so far not found a solution for this.
and i just played a bit with delphi and i could writa a programm, that starts something(i dunno how will it work and what do you need from that starter).. it is now 40Kb .. i'll play a bit later and will post some results..
it is ConsoleApp and uses WinAPI (CreateProcess with Priority flag set).. and i've found in old winSDK only four priorities.. IDLE, NORMAL, HIGH and REALTIME... and i dunno wich are usefull ..

anyway results -> later.

--and-now--------------------------

Quote:
Originally Posted by omion View Post
Grrr... Nice doesn't seem to like two backslashes in a row. Nice is not very nice. I'm really not too happy with that program, which is why I've been trying to write my own, only to learn that I'm very unhappy with C.
testing testing,.. AND:
Code:
nice -n 20 \\PC201\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid  --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input  --progress  --no-dct-decimate  --no-psnr  --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\stats e31f8a.txt"  --seek 0 --frames 5223 -o NUL "//PC201/x264farm-2/xFiles/S1p2/S1p2.avs"
nice -n 20 //PC201/x264farm/!AGENT/x264.exe --bframes 3 --b-pyramid  --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input  --progress  --no-dct-decimate  --no-psnr  --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\stats e31f8a.txt"  --seek 0 --frames 5223 -o NUL "//PC201/x264farm-2/xFiles/S1p2/S1p2.avs"
WORKS!!!!
Code:
nice -n 20 //PC201/x264farm/!AGENT/x264.exe --bframes 3 --b-pyramid  --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input  --progress  --no-dct-decimate  --no-psnr  --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\stats e31f8a.txt"  --seek 0 --frames 5223 -o NUL "\\PC201\x264farm-2\xFiles\S1p2\S1p2.avs"
Does NOT

and then after:
Code:
nice -n 20 avs2yuv -raw -seek 224022 -frames 4192 -o - "//PC201/x264farm-2/xFiles/S1p2/S1p2.avs" | nice -n 20 \\PC201\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid  --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input  --progress  --no-dct-decimate  --no-psnr  --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\224022 228213 188ff7.txt"  -o NUL - 576x432
nice -n 20 avs2yuv -raw -seek 224022 -frames 4192 -o - "//PC201/x264farm-2/xFiles/S1p2/S1p2.avs" | nice -n 20 //PC201/x264farm/!AGENT/x264.exe --bframes 3 --b-pyramid  --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input  --progress  --no-dct-decimate  --no-psnr  --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\224022 228213 188ff7.txt"  -o NUL - 576x432
WORKS!!!!
Code:
nice -n 20 avs2yuv -raw -seek 224022 -frames 4192 -o - "\\PC201\x264farm-2\xFiles\S1p2\S1p2.avs" | nice -n 20 \\PC201\x264farm\!AGENT\x264.exe --bframes 3 --b-pyramid  --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input  --progress  --no-dct-decimate  --no-psnr  --no-ssim --qcomp 0.6 --cplxblur 20. --qblur 0.5 --qpmin 10 --qpmax 51 --qpstep 4 --ipratio 1.4 --pbratio 1.3 --fps 30000/1001 --pass 1 --stats "temp\224022 228213 188ff7.txt"  -o NUL - 576x432
Does not

maybe.. it is the best way.. to change those slashes back?
__________________
Rule Number 6: Concentrate!!!
(c)Hercules, Disney

"I like to build planes.... in the air" (c) some ADV.

tutorials
How to Setup agent-based encoding with x264farm (the easy way)
SpAwN_gUy is offline   Reply With Quote
Old 14th March 2007, 00:17   #193  |  Link
omion
Registered User
 
omion's Avatar
 
Join Date: Nov 2003
Location: San Diego, CA
Posts: 325
@SpAwN_gUy:

So it looks like everything works when using forward slashes rather than backslashes? That's good to know. It's looking like it won't matter for much longer, though.

I was just contacted by foxius (at least, that will be his name when his 5 days are up ) who is making a nice alternative to the nice.exe program I am currently distributing. It will be a drop-in replacement, and should weigh in at ~50KB.

I can't say for sure, but it looks like it should be done soon (depending on how many more unreasonable requests I make )
__________________
"We demand rigidly defined areas of doubt and uncertainty!" - Vroomfondel, H2G2
omion is offline   Reply With Quote
Old 15th March 2007, 07:29   #194  |  Link
omion
Registered User
 
omion's Avatar
 
Join Date: Nov 2003
Location: San Diego, CA
Posts: 325
All right! I finally updated nice.exe!
(or rather, foxius updated it and I took credit for it )

Get it here (source). I also updated the first post's link.

Awesome new features:
  • If the program's extension is "exe", "bat", or "com", you no longer need to specifically write the extension. (This approximates the shell's handling of extensions)
  • Network paths work without nasty hacks
  • It's MUCH smaller than the old version, which had to have all the cygwin libraries dragged along with it.

MAJOR props to foxius! Thanks a bunch!
__________________
"We demand rigidly defined areas of doubt and uncertainty!" - Vroomfondel, H2G2
omion is offline   Reply With Quote
Old 15th March 2007, 10:35   #195  |  Link
SpAwN_gUy
Junglist
 
SpAwN_gUy's Avatar
 
Join Date: May 2003
Location: Belarus, Minsk
Posts: 298
) .... updated to newNice ... and looked through source..
well.. it seems, like it is doing the same thing i was thinkikng of.. and made actually ...
Code:
winAPI:
CreateProcess(with parameters)
okay.. i'll try and test it.. with a1.09 ...
currently i'm doing... oops.. done HDTV-Encode.. oops.. errore again. (something strange on merging GOPs.. c1.08 andc1.09 are doing all the same..). deleted those GOPs.. redoing them..

the second pass fps ~5-6FPS.. i think this value is.. like.. very cool for HDTV-Encode ...

and again... something strange.. in log last line is "Merging 92 / 302". Deleted 3 gops (92, 93, 94).. and waiting.

btw. feature request ... can you delete unNeeded GOPs during the second-pass? (i was looking through controller's source and found a line like "it's not an easy job to find GOPs" .. or something). so i thought.. if UnNedded(ReEncoded GOPs) are deleted during second-pass, so after crash you wouldn't need to calculate something.. (or the whole thing is working different?)

and one more thing.. about "more accurate bitrate". with no additional params(overrides).. two of my encodes were +100Mb.. instead of 1460 they are 1560.. i guess i'm back to "--3thresh 0.95 --3ratio 0.2"

okay,.. i've done "pause" in .bat file... so..
Code:
Fatal error: exception Invalid_argument("Char.chr")
i dunno what does that mean, and it des not look like shellException... (i have all the temp files.. and i think i'll send those to you..)
and the file is allways like 211Mbs...
__________________
Rule Number 6: Concentrate!!!
(c)Hercules, Disney

"I like to build planes.... in the air" (c) some ADV.

tutorials
How to Setup agent-based encoding with x264farm (the easy way)

Last edited by SpAwN_gUy; 15th March 2007 at 11:41.
SpAwN_gUy is offline   Reply With Quote
Old 16th March 2007, 07:41   #196  |  Link
omion
Registered User
 
omion's Avatar
 
Join Date: Nov 2003
Location: San Diego, CA
Posts: 325
Quote:
Originally Posted by SpAwN_gUy View Post
) .... updated to newNice ... and looked through source..
well.. it seems, like it is doing the same thing i was thinkikng of.. and made actually ...
Yeah. I had the basic idea of how to do it (involving CreateProcess), but my programming skills are basically limited to Perl and OCaml, neither of which are particularly good at hooking into the OS. And C just drove me nuts.

And about me using foxius's code rather than yours, well... he contacted me first Don't worry; I'll tell you the next time I need a program that I don't want to write

Quote:
okay.. i'll try and test it.. with a1.09 ...
currently i'm doing... oops.. done HDTV-Encode.. oops.. errore again. (something strange on merging GOPs.. c1.08 andc1.09 are doing all the same..). deleted those GOPs.. redoing them..

the second pass fps ~5-6FPS.. i think this value is.. like.. very cool for HDTV-Encode ...

and again... something strange.. in log last line is "Merging 92 / 302". Deleted 3 gops (92, 93, 94).. and waiting.
That looks like something's wrong in the file itself...

Quote:
btw. feature request ... can you delete unNeeded GOPs during the second-pass? (i was looking through controller's source and found a line like "it's not an easy job to find GOPs" .. or something). so i thought.. if UnNedded(ReEncoded GOPs) are deleted during second-pass, so after crash you wouldn't need to calculate something.. (or the whole thing is working different?)
Fixed. The next version will delete the old GOP files as soon as the re-encoding is finished, unless the --keeptemp option is specified (which, I just noticed, hasn't worked since ~1.07... )

That comment about finding GOPs (around line 450 in pass2.ml) was actually referring to the fact that there could be many more GOPs done than are in the temp stats file, and how it is quite difficult to determine where to restart counting when the program is re-run.

However, the only thing deleting the unneeded GOPs will get you is extra disk space. Everything is calculated based on the working_stats#.txt files. The result is that, once the reference to the old GOP is overwritten by the new GOP file name, the old GOP file might as well not exist any more. Nothing in the program references old GOP files, so they don't have any effect on anything.
This means that there is no reason not to delete them, but it won't help the program if they are deleted.
In either case, the whole temp directory is cleaned up at the end of the second pass, so the final result is the same no matter what files are in there.

Quote:
and one more thing.. about "more accurate bitrate". with no additional params(overrides).. two of my encodes were +100Mb.. instead of 1460 they are 1560.. i guess i'm back to "--3thresh 0.95 --3ratio 0.2"
Hmm... I'll have to look into it a bit more. Ratecontrol is quite a subtle thing, and it's fairly easy to do wrong.

I probably have a plus when I should have a minus or something. Due to the self-correcting nature of ratecontrol, just about every bug in it will result in something that is a bit off, but not by much.

Just curious: what are you using to measure bitrate? The bitrate that x264farm (and x264) use are raw AVC bits. The container always adds overhead to that. It shouldn't be 8%, though, so I think something's wrong anyway.

Quote:
okay,.. i've done "pause" in .bat file... so..
Code:
Fatal error: exception Invalid_argument("Char.chr")
i dunno what does that mean, and it des not look like shellException... (i have all the temp files.. and i think i'll send those to you..)
and the file is allways like 211Mbs...
Where does it say this? Right when it fails to merge the files? If so, that is either a bug in my Matroska-parser or a bug in the Matroska file itself (probably the parser, but I don't know why it hasn't popped up until now)


(apologies for any horrible grammar or anything in this post. My brain is not working properly right now. Too much caffeine... need more sleep...)
__________________
"We demand rigidly defined areas of doubt and uncertainty!" - Vroomfondel, H2G2
omion is offline   Reply With Quote
Old 16th March 2007, 09:11   #197  |  Link
SpAwN_gUy
Junglist
 
SpAwN_gUy's Avatar
 
Join Date: May 2003
Location: Belarus, Minsk
Posts: 298
Quote:
Originally Posted by omion View Post
And about me using foxius's code rather than yours, well... he contacted me first Don't worry; I'll tell you the next time I need a program that I don't want to write
don't worry, i'm beginning to be more opened to peaple,.. but not all the time ... i gust fing your progam very interesting, and i'm trying to help.. thats all

Quote:
That looks like something's wrong in the file itself...
....
Where does it say this? Right when it fails to merge the files? If so, that is either a bug in my Matroska-parser or a bug in the Matroska file itself (probably the parser, but I don't know why it hasn't popped up until now)
well,.. it shows right after writind to log file line "Merging 91 / 302"
and, i've muxed with my bare hands (mkvmerged) all those .mkv-files.. some of them were "unKnown format" (i guess one of those zero-bytes file, i have in _temp), and (i guess i forgot some of files, but.. ~godd mkv.. (it became 1 second less)) the end file is good,.. (thise 91,92,93 - were good in the end).
so i'm sending you those files.. via e-mail..


Quote:
Fixed. The next version will delete the old GOP files as soon as the re-encoding is finished, unless the --keeptemp option is specified (which, I just noticed, hasn't worked since ~1.07... )
waiting ..

Quote:
Just curious: what are you using to measure bitrate? The bitrate that x264farm (and x264) use are raw AVC bits. The container always adds overhead to that. It shouldn't be 8%, though, so I think something's wrong anyway.
GordianKnot-calculator (aac-sound goes as vorbis in "Interleaving & AVI-Overhead") or Calculator in meGUI.. (as far as meGUI has "GUI" and Profiles,.. and controllerGUI can import meGUI Jobs... so, generating a comLine is not a problem to me.. ..)

btw. minasan, how would you like to start controller? invisible and all "usefull info is displayed in GUI"(as meGUI does) or just run a pregenerated comLine, while showing DOSBox(like GKnot does)?

yeah,.. and something like a BUG,.. i think.. after "resuming" on second-pass,.. the info(that periodically is shown..) does not show "Current GOP" (column "Now") just "x"..
__________________
Rule Number 6: Concentrate!!!
(c)Hercules, Disney

"I like to build planes.... in the air" (c) some ADV.

tutorials
How to Setup agent-based encoding with x264farm (the easy way)

Last edited by SpAwN_gUy; 16th March 2007 at 09:20.
SpAwN_gUy is offline   Reply With Quote
Old 19th March 2007, 09:22   #198  |  Link
foxius
Registered User
 
Join Date: Mar 2007
Posts: 17
Finally, I have a voice =)

x264farm is working like a charm here, the only bug I've found so far is that sometimes I get a DELFIRST from an agent and it doesn't resolve until I kill the agent, clean his temp folder and restart. I think, it would be more convenient if the agent could clean the broken files itself.


I've been using x264farm for a week now and it saved me lots of hours. I have 7 agents:

1. 1x working on the controller, Pentium D 820 2.8GHz, gives ~7-8fps in the second pass

2. 3x on the same 100mbit switch:
- Celeron D 2.6GHz, ~3-4fps
- Celeron D 2.6GHz, ~3-4fps
- Pentium 3.2GHz with HT, ~4-5fps

3. 3x on another switch over 100mbit uplink:
- Pentium 2.8GHz with HT, ~3fps
- Pentium 2.8GHz with HT, ~3fps
- Pentium 2.8GHz, ~3fps

Using agent based encoding gave me like 4 times speed up. Obviously, I'm losing 1-2fps because of the slow uplink. Sending 2 jobs to a CPU with HT increases the performance only by 0.3-0.4fps while putting more load on the network.
foxius is offline   Reply With Quote
Old 19th March 2007, 19:09   #199  |  Link
omion
Registered User
 
omion's Avatar
 
Join Date: Nov 2003
Location: San Diego, CA
Posts: 325
Quote:
Originally Posted by foxius View Post
Finally, I have a voice =)

x264farm is working like a charm here, the only bug I've found so far is that sometimes I get a DELFIRST from an agent and it doesn't resolve until I kill the agent, clean his temp folder and restart. I think, it would be more convenient if the agent could clean the broken files itself.
Well, DELFIRST is not really a problem in itself. It comes about when an agent finishes a job right before another job was being worked on, and it discovered that the first frame on the second job won't be an I frame. DELFIRST really just means "delete first GOP in the job, when it gets returned" and it will be taken care of when the agent returns the job.
It turns out that dealing with this case correctly is quite difficult, so I added it to the output so I could properly debug it.

If the job is never returned, or if there is a constant cycle (the same frames get re-encoded over and over) then that's a problem.

It is, however, extremely odd that you need to clean out the agent's temp folder manually. The agent has no memory of any previous jobs when it gets restarted. When a new request is made, the agent picks completely new temp files for the encoding. It wouldn't even know where to check for the old temp files, as all the file names are generated randomly.


@SpAwN_gUy:
I didn't find anything wrong with the MKV files you sent me. I checked them with the official mkvinfo tool and they parsed correctly. I used the same Matroska-decoder as in x264farm to merge them and everything went well... so I don't know where the problem is. Are you still able to recreate it? If so, I can send you a new build which should at least give you a more descriptive error message.

And sorry about the delay. I thought I would get all this stuff done last week, but I got sick. And I play too much Oblivion. (Maybe I got sick from playing Oblivion...?)
__________________
"We demand rigidly defined areas of doubt and uncertainty!" - Vroomfondel, H2G2
omion is offline   Reply With Quote
Old 19th March 2007, 20:06   #200  |  Link
foxius
Registered User
 
Join Date: Mar 2007
Posts: 17
Quote:
Originally Posted by omion View Post
If the job is never returned, or if there is a constant cycle (the same frames get re-encoded over and over) then that's a problem.
Yeah, that's exactly the problem I'm having sometimes. One agent is stuck and others are just waiting.

Quote:
Originally Posted by omion View Post
It is, however, extremely odd that you need to clean out the agent's temp folder manually. The agent has no memory of any previous jobs when it gets restarted. When a new request is made, the agent picks completely new temp files for the encoding. It wouldn't even know where to check for the old temp files, as all the file names are generated randomly.
Perhaps it would work even without removing the temp files. I'll try not to delete them next time.

I'm not sure if it's possible to detect a DELFIRST dead loop, but I'd suggest putting a limit to consecutive DELFIRST messages received from an agent. If it's reached, then the controller assumes that the agent is stuch and sends a signal to drop the job and restart. Of course, this is only a quick&dirty workaround. Can't think of anything else.
foxius is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 12:17.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.