RB Farm v1.7 - server farm for faster DVD Rebuilder encodes

[djackson]

Thanks Quantum. Reinstalled AVISynth. All now works.

[Mutant_Fruit]

i'm just finishing setting up my 2 computer network.

An overclocked 2500+ and a standard speed 1800+. Lets test this baby out!

[Mutant_Fruit]

Yeah, there is a biggish performance hit when you use mapped drives.

Is there any way to share the drives/files across a network without having to map them using windows? Even mapping my local drive to M: and then (using the computer with the drive physically attached to it) i try and use DVD-RB to prepare, pointing it towards the M:, it goes slowly.

[Mutant_Fruit]

I had an idea. If RBFarm could be coded so that it can read files from an FTP server.

Basically, i just download a free FTP server app, then use the server to set the address of the D2VAVS folder to be something like ftp://192.168.0.1/D2VAVS and then i make both computers look to that for the folder.

That would stop the slow access with the drive remapping in windows. I hope i explained that well.

[Joergen]

FTP doesnt support random access in the same way true drive mapping does. Also for localhost drives, as mentioned in this tread, use the subst command instead. It maps your drive to a new letter with absolutely no slowdown (old optimized DOS command).

[Mutant_Fruit]

I didn;t know that about FTP, thats that idea down the chute.

And i didn't realise that the subst command meant no slowdowns. I thought it was because he was too lazy to map the drives normally (or wanted them mapped a different way for something else). I'll give it a shot now.

Thanks for the tip.

[Mutant_Fruit]

erm... explain exactly how i use subst to set up my D: to share between 2 computers. This computer i'm on, and one other.

[Joergen]

Oh, I thought since you mentioned prepare or rebuild being slow (which it is if you use the network mapping on localhost drives) you wanted a faster method, which subst provides.

You cant subst via a network. So the best way is to say subst the D: of PC#1 to O: and setup O: as the paths in dvd-rb, then map D: of PC#1 as O: on PC#2 (network mapping).

Now when you prepare and rebuild on PC#1 it will be as fast as if the O: was a real drive.

Preparing or rebuilding OVER the network will of course be slow since it's all about HD speed, and you'd need a gigabit network to be fast enough (and even then access times are several times as slow).

[Mutant_Fruit]

I understand now. subst is only used on my main computer. I thought subst was supposed to be used on all the computers, and i just couldn't get it to work on my second PC. Bah, i'm an idiot.

I got subst workin on this PC, and the preparing time is back down to 7mins, nice and zippy. Thanks for the pointers. I'm getting the hang of this now :) Great proggy btw.

Is there any way to donate via bank transfer?

[Joergen]

Paypal is sad in that most people in the nordic countries and most of europe dont use credit cards (like me) because you can always go around the corner to the ATM machine. And many shops dont even accept them due to a % going to the card company and cause of fraud :(

[Mutant_Fruit]

How do i go about setting up RB Farm and my computers to encode multiple movies in one night. Is "Batch mode" supported in RB Farm 1.6?

I tried running multiple instances, but EclCCE won't allow that. I don't want to set it up incorrectly, and end of overwriting each finished show as it starts the next.

[Mutant_Fruit]

bump. any ideas? From the looks of it, this feature isn't supported yet. But maybe i'm just stupid...

[Joergen]

Sorry, I dont think its supported. I've never needed a batch job myself as I like to play around with settings and tweak the output.

[Mutant_Fruit]

Ah well, looks like i have to wait for it to be supported.

In the mean time, could an option be added to turn off the farm computers when no more parts are available to encode. No point in my second computer being on all night when it finishes encoding about 1am...

[quantum]

Updated to version 1.7. This build adds batch processing functionality.

I wasn't going to bother adding this until I realized you can't do a batch job (unless I'm mistaken) in DVD RB unless you do one click mode. This means DVD RB will run a prepare phase wether you like it or not and any customizations you make with ECL Tweaker or ECL Optimizer, etc., are overwritten.

So now you can run the prepare phase on multiple jobs, make adjustments, then queue up the jobs in RB Farm. Even with one PC, which is all I use for encoding, this is an improvement.

This is not heavily tested so I wouldn't be surprised if some things have problems, especially the 'auto rebuild' feature, which won't work at all with multiple jobs.

[Joergen]

Sweet, Fruit got his wish! :cool:

[Mutant_Fruit]

LOL, i didn't expect that to happen.

I think its time i PM'ed the writer of DVD-RB and RB Farm and try and figure out some kind of bank transfer, as i really couldn't be arsed using paypal. I probably will be pretty busy for the next month or so (REALLY important exams coming up, that will decide the rest of my life!) so i won't be using my computer much over the next month or so (til the 25th of june, at 11:30a.m.).

Then i'll be back here, testing the latest version of RB and Farm, and organising some donations.

Now... if only i could recover that crashed harddrive...

[Joergen]

quantum: I know this might be impossible but please consider it.

I explained in the 30+ bug thread how a chunk got canceled via CCE first thus leaving a short .m2v.

Would it be possible to make RBFarm detect when CCE is canceled prematurely/first?

Or perhaps an option to check that all chunks are within a range and large enough, perhaps based on the average bitrate in the .ecl for each chunk?

[quantum]

Of course it's possible, but the problem won't happen if you cancel via RB Farm first. The only way to cause it is if you abort CCE first (just don't do it) or have a crash.

If you have a crash then you should know it. At that point get in there and clear it up manually. How many crashes do you have?

I don't know if I'll get to this since it's never been a problem for me, and it's not trivial to code.

[Joergen]

I dont crash (ever, knock knock knock) but somebody had canceled the job prematurely manually on one of the PC's while I was away.

If the PC would crash, it would reboot thus RBFarm would not write the finished file for that chunk.. no probs there.

[quantum]

I thought you were referring to the crash scenario. Regarding the manually aborted CCE scenario, now that I think about it, RB Farm would have a very hard time figuring out that an m2v was aborted versus a regular CCE finish. I would have to get into figuring out how many frames are in the m2v and compare to how many are reported in the AVS, both of which I have no idea how to do. In any case it would be hard work, possibly very hard work, and I'm sure I won't go down that road considering how rare the event should be.

Have you used the 'minimize CCE' checkbox in RB Farm? That should reduce the chances of accidentally closing the program. I use that myself.

Otherwise put a 'do not touch' sign on the PC. ;)

[winny]

Thanks for the added function Quantum, this is a very welcome surprise. I will start testing immediately!

[NobbyNobbs]

It looks like the link is not working properly at the moment. (The same goes for ECL Tweaker)

[acido]

@quantum
just a little problem i'm having with RB-FARM 1.6 (and 1.7) with latest builds of dvd-rb (0.51).
Yesterday i PREPARED with dvd-rb then started rb-farm on 2PCs with the following options :
- minimize cce
- auto encode
- auto rebuild ( only on the pc where i did the prepare )
.
First thing i noticed is that when rb farm started it did not miimize the DVD-RB window as usual, then at the end of the CCE steps, rb-farm was unable to auto rebuild and a popup window indicating it was unable to find dvd-rb window appeared (sorry i cannot remember the exact syntax).
Anyway after starting the rebuild manually on dvd-rb all went ok.
.
Are you getting the same?

[d62ks821]

It would be a fairly simple matter to remote administer the farm if rbfarm had the capability to accepted all options as command line params. For instance, if I launch rbfarm via telnet, pcexec, ssh... rbfarm with start up with the rebuilder.ecl path blank, and as such does not start processing.

[d62ks821]

Originally posted by chadp1a
Followed your advice on the sluggish network problems and it worked... thanks...

getting cce speed of 1.9+ on both pc's now :)


I have a similar problem. 2 nearly identical PC's the PROC on main is pegged at 100% while the PROC of the second only raises to 50-70%. Here is my config (main--switch--second):

ATH 1700+/512MB DDR ----- 100/10 switch ----- ATH 2100+/512MB SDRAM

Things I tried to debug this problem:

1) Stopped farm on main. Only slight increase in PROC utilization on second.

2) LAN utilization is only at 10% (6% main->second, 4% second->main), so i didn't think net connection/switch was a problem, but none the less ran some file copy experients. I could achieve 160% utilization by simultaneously copying a lG file main->second adn second->main. So clearly I'm at 100MB full duplex.

3) verfied that CCE is running at IDLE prority on both machines, and on the main machine CCE is consistantly running at around 97% of PROC utilization. ...so dont think problem is main side serving.

4) suspect problem was some drive mapping overhead, so mapped source rip and working drives locally on main. and GUESS WHAT!!! now PROC utilization on main is 50-70% as well! SO, I tried mapping the source rip drive and subst the working drive and PROC utilization was back to 100%. The inverse (subst the source rip and mapping the working) also was 100%.

CONCLUSION: Problem has nothing to do with my LAN. Problem appears to be some inefficiency (or extra wait states) with Windows XP when there is simulanteous activity on two different mapped drives.

[d62ks821]

Originally posted by d62ks821

CONCLUSION: Problem has nothing to do with my LAN. Problem appears to be some inefficiency (or extra wait states) with Windows XP when there is simulanteous activity on two different mapped drives.

One additional experiment. I move my source rip to the same physical HD as the working directory so I could run with only one mounted drive. AND running the main PC farm through 1 mapped drive and the second PC through 1 mapped drive, still resulted in poor PROC utilization (50-70%) on both PC's. I'm stumped.

[Joergen]

I've also noticed that since upgrading to a more recent .dll (decodefix110) and rbfarm 1.6 the fastest PC is only doing 60-70% of cpu time while with the old configuration of rbfarm 1.0 and previous mpeg2dec .dll it did 90-100%.

I'll try downgrading. I kept the 1.0 rbfarm somewhere.

[quantum]

Originally posted by acido
just a little problem i'm having with RB-FARM 1.6 (and 1.7) with First thing i noticed is that when rb farm started it did not miimize the DVD-RB window as usual, then at the end of the CCE steps, rb-farm was unable to auto rebuild and a popup window indicating it was If this was with 1.7 then I'm not surprised. Since I added the batch job capability, I haven't tested the 'auto rebuild' feature. RB Farm can't rebuild if you have more than one job, and maybe not even with one job (untested by me). I'll look at it when I have time.

The only way I can think of to get around the multiple-jobs and auto rebuilding issue is to accept only saved projects (rbd files) as input, instead of pointing to rebuilder.ecl.

If your problem is with 1.6, then it was likely a random failure. The way I coded RB Farm to remote control DVD Rebuilder is likely not to work 100% of the time.

[gnode]

Originally posted by acido
@quantum
just a little problem i'm having with RB-FARM 1.6 (and 1.7) with latest builds of dvd-rb (0.51).
Yesterday i PREPARED with dvd-rb then started rb-farm on 2PCs with the following options :
- minimize cce
- auto encode
- auto rebuild ( only on the pc where i did the prepare )
.
First thing i noticed is that when rb farm started it did not miimize the DVD-RB window as usual, then at the end of the CCE steps, rb-farm was unable to auto rebuild and a popup window indicating it was unable to find dvd-rb window appeared (sorry i cannot remember the exact syntax).
Anyway after starting the rebuild manually on dvd-rb all went ok.
.
Are you getting the same?

Same problem here on my farm!

[acido]

Originally posted by quantum
If your problem is with 1.6, then it was likely a random failure. The way I coded RB Farm to remote control DVD Rebuilder is likely not to work 100% of the time.

@quantum
Tried another time,
problem is solid with 1.6, i think something has changed in DVD-RB (for sure with 0.51b) and RB-FARM is no longer able to interface it.
As you can see gnode is having the same trouble, i'm not the only :) ...

[d62ks821]

Originally posted by Joergen
I've also noticed that since upgrading to a more recent .dll (decodefix110) and rbfarm 1.6 the fastest PC is only doing 60-70% of cpu time while with the old configuration of rbfarm 1.0 and previous mpeg2dec .dll it did 90-100%.

I'll try downgrading. I kept the 1.0 rbfarm somewhere.

How did your experiment go? I still have been unable to determine why I'm getting as little at 50% proc utilization on the networked farm PC. Other things I tried:

1) once CCE is running on the networked machine, I kill rbfarm on both PC's. Still have poor PROC untilization on networked PC. I don't think rbfarm is the problem.

2) ...next I kill the CCE on the main PC, so the only thing left running is CCE on the neworked PC. Still have poor PROC untilization on networked PC.

[gnode]

I have an 8 machine farm and strangely I can get 100% cpu utilization by running the xp performance monitor in the background on the slave machines. If I bring the performance monitor to the foreground the cpu utilization drops down to the 75% range. If I highlight the farm app, so that the performance monitor goes to the background, the cpu utilization goes back to 100% but not right away. Sometimes it waits for a new rb process to start before it goes back to 100%.

Give this a try and see if you can reproduce my results.


BTW my farm is a mix of amd and intel machines and the result is similar on both types.

[d62ks821]

Originally posted by gnode
I have an 8 machine farm and strangely I can get 100% cpu utilization by running the xp performance monitor in the background on the slave machines...

Give this a try and see if you can reproduce my results.


I presume you are referring to the performance tab on the task manager. Task manager runing, not running, forground, or background makes no difference.

I noticed that one of my disks was making alot of thrashing noise when I run via mapped network drive. So to try and eliminate the thrashing (and presumably the associated IO wait time that is likely causing the poor PROC utilization), I tried these things:

(1) I disabled system swap file, to eliminate it as a possible bottle neck. No difference. So re-enabled.

(2) Set swap file to fixed size of 2X my DRAM (i.e., 1GB). And then defragged boot disk. No difference.

(3) Defragged disk where my working directory resides. No difference.

[gnode]

d62ks821,
Are you saying that you repeated my experiment of having the task manager performance window running, not running etc. and saw no change in cpu utilization?

BTW, I'm using a 3com managed switch and I set cce on the master to idle as mentioned in page 3 of this thread.

gnode

[d62ks821]

Originally posted by gnode
Are you saying that you repeated my experiment of having the task manager performance window running, not running etc. and saw no change in cpu utilization?
correct. no change.

Originally posted by gnode
BTW, I'm using a 3com managed switch and I set cce on the master to idle as mentioned in page 3 of this thread.
master CCE is set to idle priority. And i'm using a netgear 10/100 switch. (however, see my ealier post, i doubt the network connection is the problem because I can reproduce poor PROC utilization on the master running by itself, with no slaves, by simply network mapping the working drive locally rather than using subst command)

[d62ks821]

Originally posted by d62ks821
I'm stumped

Ok, Major breakthrough here! By moving input and output files around, I was able to isolate problem to .mpv file output by CCE SP v2.50. About 30 seconds into the writing of the .mpv file, the disk starts to thrash unexpectedly. Looking around in the CCE menus I found: "/Options/Misc/Use Overlapped Output" which is checked. If I uncheck it, proc utilization gets much better and disk stops thrashing.

CCE on master outputing to network mapped drive on master:
With overlapped checked proc utilization is 50-70%
With it unchecked, pegged at 100% !!!

CCE on slave outputing to network mapped drive on master:
With overlapped checked proc utilization is 50-70%
With it unchecked, at 90% !!! (still don't undertand the 10% loss)

To clear this check box by default, set the following in the system registry:
"HKEY_CURRENT_USER\Software\Custom Technology\Cinema Craft Encoder SP\overlapped = 0"

[gnode]

The version of CCE I'm using (2.6x) doesn't have this setting. Disabled by default? I'll check if adding a key to the registry makes a difference.

Gnode

[gnode]

I changed to another 3com switch and now I get 100% cpu utilization on all 8 hogs on the farm!!


Still have the error when it tries to do auto rebuild!

Gnode

[lrosado]

When I run RBFARM on the second PC I get an error that states that it could not get the information. CCE Loads up but with the error and doesnt encode. I think it says something like "GdipLoadImageFromFile Failed :3". What could I do to get this fixed?

[lrosado]

anyone????

[gnode]

Did you do the map network drive correctly? Are the movie and working folders on the first machine set to sharing?

[lrosado]

Nevermind, go it to work, I thought I had installed the avisynth and I never did. I installed everything else but that, stupid me. Thanks to those who replied.

I got cce to do 3.0 on PC#1
I got CCE to do 2.0 on PC#2

thats 5x combined speed, awesome little program.

It did get stuck at the auto rebuild phase(only selected on the main PC), said it could not find the rebuilder window or something like that.

[Mutant_Fruit]

best not to use auto-rebuild. They way its coded, it doesn't always work due to the way it had to be coded. The best thing to do is save each project in DVD-RB, and after its compresed, rebuild it manually.

[Uppercuts]

I set up rbfarm on all three of my PC's. It works fine on the main PC but when I try to encode on client machines it gives me the "Can't Get File Attributes" error. Is there something i am doing wrong? Help will be greatly appreciated. Thanks...

[lrosado]

make sure all working paths (path to d2vavs) are the same on all rbfarms and that both the source and working directory is on the mapped drive. and that avisynth v2.54 is installed on all three machines along with CCE along with ECLCCE, if needed, on all three machines.

Infiniterb wrote

I did the prepare process on one PC, and have that drive mapped to Z:

RB's paths are the folllowing:
The source is Z:\JurassicPark
The Destination is Z:\VDWRKDIR

For DVD RB Farm:

On PC ONE

D2VAVS Path: Z:\VDWRKDIR\D2VAVS
ECLCCE path: C:\Program Files\ECLCCE\eclcce.exe

On PC TWO

D2VAVS Path: Z:\VDWRKDIR\D2VAVS
ECLCCE path: C:\Program Files\ECLCCE\eclcce.exe

You would just do the same for PC#3

[d62ks821]

Originally posted by gnode
I changed to another 3com switch and now I get 100% cpu utilization on all 8 hogs on the farm!! Gnode
Have tried several things, and still can't peg CPU utilization at 100% on slave pc's.

Netgear router switch - 85-90%,
3Com switch - same.
no switch (crossover) - worse (65-75%), likely due to the fact that for some reason I can only get windows to run half duplex this way, despite setting both computer at fixed 100 FD in the driver advanced settings.

Using Intel 8255x based ethernet adapters in both machines.

[gnode]

Originally posted by d62ks821
Have tried several things, and still can't peg CPU utilization at 100% on slave pc's.

Netgear router switch - 85-90%,
3Com switch - same.
no switch (crossover) - worse (65-75%), likely due to the fact that for some reason I can only get windows to run half duplex this way, despite setting both computer at fixed 100 FD in the driver advanced settings.

Using Intel 8255x based ethernet adapters in both machines.

Have you tried changing the various parameters on the ethernet cards, such as flow control, etc.? I set all parameters on all machines the same.

When I had some intel ethernet cards in some machines and 3com cards in others I had problems. Things improved when I went to all 3com cards and a 3com switch.

Also, one of my slaves is an Nforce2 machine with an addon pci ethernet card and for some unknown reason it has trouble staying at 100% cpu utilization. My master is also an Nforce2 machine. All the other machines both AMD and Intel all pretty much peg at 100%.

For kicks I tried my 8 port etherfast switch -- poor cpu utilization all around.

[wink]

For my problem, I came to the conclusion that XP is the culprit. At first I use 2 computers, one 2k, one xp with sp2 (beta) the cpu utilization on both computers are high. As I add my laptop (XP no service pack) to the equation, the cpu on laptop is around 60-70%. I tried change the file serving from 2k to XP, both low cpu utilization. But when I update xp to sp2 (beta), the cpu usage jump up to around 95% during encoding process. Just want to let you guys know. As always, BACK UP your system first just in case MS screw up ;)

[old-hack]

I've just successfully used RBFarm 1.7 and am really impressed.

I thought I'd throw out a couple of things that I found were not totally obvious and clear about configuring the source computer and Farm computers - especially for a new user to RBFarm.

Please, no flames about reinterating what a lot of you may think was obvious.

1. On the source computer where the files are "prepared", you must prepare the DVD files using a mapped drive to the DVD source
"VIDEO_TS" folder. Do not prepare your files using the drive letter of your hard drive on the source PC. This mapped drive letter must be the same in the "source path" box in DVDRebuild on both the source PC and the Farm PCs.

2. You must create a mapped drive to the "working folder" created on the source PC. This mapped drive letter will be the drive letter you put in the "working path" box in DVDRebuilder on both the source PC and the Farm PCs.

3. The DVD Rebuilder directory in RBFarm is the local PCs path to DVDRebuild for the Farm PCs. You do not create a mapped drive for this path.

4. The "Rebuilder ECLs" path in RBFarm is the "D2VAVS" folder beneath the drive letter mapped to the "working folder" on the source PC. When you browse the mapped "working folder" drive, you will see the "D2VAVS" folder. Once you select this folder and press "OK", the correct path will appear in the "Rebuilder ECLs" path in RBFarm. The mapped drive letter\"D2VAVS" path must be the same on all Farm PCs.

As a suggestion and for clarity, I think the field labeled "Rebuilder ECLs path" in RBFarm shoud say "Source D2VAVS path" or "Mapped Source D2VAVS path".

Thanks.