RB Farm v1.7 - server farm for faster DVD Rebuilder encodes

[quantum]

HERE (http://www.connforum.com/rbfarm.zip) is a link to the current version of RB Farm. The zip file size should be 25,585.

This thread describes the development and testing of RB Farm, an assistant program to DVD Rebuilder. RB Farm allows multiple PCs to work together on a network to perform a faster CCE encode than could be performed with a single PC.

1.7 - Added batch processing functionality. You can queue up multiple jobs and they will be processed in order.

1.6 - Upgraded auto encode functionality. You no longer need to press the encode button on a single PC to start the encode process. This should allow one click functionality on a farm. You should be able to press "prepare" in DVD Rebuilder and RB Farm will automatically do the rest.

1.5 - Fixed some problems with "rebuild automatically"

1.4 - Added auto rebuild
The "auto rebuild" check box will cause the current instance of RB Farm to attempt to rebuild the project after encoding using DVD Rebuilder. You should only select this option on the PC where DVD Rebuilder is installed and where you ran the prepare phase. DVD Rebuilder should be running with the settings for the current project correctly chosen, i.e., audio streams deselected, etc. The mode should not be one click mode, and "suppress warning prompts" should be selected.

1.3 - Added auto encode functionality
The "auto encode" check box will cause the current instance of RB Farm to periodically detect if a job is ready to be encoded in the d2vavs directory. If a job is ready, RB Farm will automatically join in and process the next available chunk.

1.2 - Added support for CCE Basic. You have to type in the EXE file name manually on the "path to ECLCCE" line.

1.1 - Fixes CCE 2.5 support

[Joergen]

Rocking :)

Mirror:
rbfarm.zip (http://koti.mbnet.fi/haupt/rbfarm.zip) 1.7 25KB

[EddieTH]

I'm trying out a modified version of this now. I have 3 PCs to encode with, but the CPU and RAM vary widely between the 3 (P4 2.4GHz/768MB DDR, Celeron 1.7GHz/128MB DDR, & Athlon 1GHz/256MB SDRAM). Since taking 1/3 for each PC would actually take longer because the fastest machine encodes at more than 5x the speed of the slowest, I did a test encode with one of the segments and compared the speeds on each of the PCs. Then I calculated what percentage of frames they should each encode to get done at roughly the same time. This would be the fastest I could possibly encode with this configuration. Finally, I determined how many segments to encode with each PC based on the frame count.

[quantum]

I thought more on this, but I didn't post anything simply because I lost interest. To me, I'm not that speed hungry. I don't mind queueing up batch jobs and waiting overnight. But with said, I'll offer a bit more:

The latest version of CCE does not have the memory leak problem that has plagued previous versions when loading multiple AVS files (see the CCE forum). I downloaded the trial version and indeed it looks fixed. Thus you should be able to simply load the ECL file into each instance of CCE (on separate PCs), then trim out half (or whatever) jobs in CCE (with ECLCCe) and encode. This would eliminate all the nonsense steps listed above.

Additionally, I figured I could code up a quick program in a few hours or so to take jdobbs rebuilder.ecl file and feed each chunk to eclcce on each machine. This would have the benefit of automatic load balancing and reduce the manual steps down to nothing. Maybe I'll get to this if someone else doesn't do it, but as jdobbs said, it's probably better to wait until the engine is 100% stable before worring about things like this.

[EddieTH]

I agree, it's not a big priority, especially since my second machine only encodes at about 1/3 the speed of the first, and the third machine is significantly slower than that. I also noticed that the speed for each of the remote encodes dropped by about 10% when they were all running while the local encode slowed down by around 20%. Even when the third machine finished (it got less than planned because of the size of individual segments), the other 2 encodes didn't get that much better speed.

I'll probably play around with this more since I have a lot of time to do it, but I probably won't do anymore 3 part encodes since the gain from the last machine is minimal. My main interest is in seeing how much performance DVD-RB can manage with multiple PCs, which I would say is a fairly significant amount with the right setup, but not enough to be worth the trouble in many cases.

[Joergen]

You could use the slowest computer to serve the source and the faster ones to compress. With DMA 100mbit LAN the lan transfer itself shouldnt eat up more than 1-5% of CPU time.

But if you have old 10Mbit LAN cards the CPU usage might be 20-50% depending on the chip and driver and speed of transfer.. but mostly those cards are history.

[EddieTH]

I probably could, but there's another issue with the slow machine. It's a laptop. I'm not real excited about the idea of heating it up for that long. I have another machine I'm building with a case and mobo that was given to me. It will probably end up being about the same speed as the first one (or maybe a little faster), so I may try out 3 PC encoding again when it's finished.

[Joergen]

I have 3 desktop PC's in this room and the makings of two more stored away, I only need 2 ATX cases with PSU. If somebody writes a handy extension to dvd-rb or jdobbs gets around to supporting this idea I'll surely slap together those puters :)

[quantum]

Maybe I will fire up something if you two want to test it. Shouldn't take too long..

[quantum]

If you two want, send me a PM with an email address where I can send beta copies. I can't post attachments here. I should have something ready tonight.. (famous last words of every programmer)

[EddieTH]

If you make it I'll certainly put it to work. I work from home so I'm in front of my computers for a great deal of the day. It's nice being able to combine hobbies and work. :cool:

[Joergen]

Sounds great, but could you host it somewhere instead so we can dl. :o

[quantum]

I completed a very rudimentary prototype. It works but it's too basic to post and I expect problems so I'd like to get an email correspondence going with testers to avoid cluttering up jdobbs forum.

@EddieTH I sent you a copy via email.
@Joergen I sent you a PM with the location and details.

Anyone else interested in testing this PM me (preferrably with an email address) and I'll send you a copy.

[Joergen]

Any news here? I'll be trying the first version now with a disc and two puters.

[quantum]

I guess it's been tested enough.

HERE (ftp://ftp.connforum.com/rbfarm.zip) is a link to the current version of RB Farm. I'm not sure how long I'll keep it hosted there, but considering how small it is, I guess I'll keep there for a little while. If jdobbs or doom9 consider it worthy, maybe it will be hosted in a more permanent spot.

Big thanks to EddieTH for testing and debugging. Good testers are hard to find. Of course big thanks to jdobbs for all his great work.

Instructions are in the readme. Comments are welcome.

[Joergen]

Rocking! Sorry for not being able to test it earlier.

Hmm thats interesting, the error 9 seems to come from removing streams in shrink. So I'll set the targetsectors higher and overshoot, then use shrink to remove the huge-hiney subs.

[Joergen]

The WORKING and SOURCE path, it would seem, really limits the use of RBFarm for most (since how many people have the source computer with a path that is so far down the alphabet that the other PC dont have it reserved.)

It might be obvious to some but not all:

TIP:

1) On the source PC (the one that has the dvd files, and want to run DVDRB on and do the PREPARE), go to the network neighborhood and map the drive (ie D: ) to H: and use H:\ as the WORKING PATH in DVD-RB.

2) Then on PC#2 map the same drive (D: @ PC#1) as H: on PC#2.

There you have the same WORKING path on both PC's easily.

You need to do the same to the source path, so placing the source in ie D:\MOVIE\VIDEO_TS on PC#1 and then using H:\MOVIE\VIDEO_TS as the source. And of course H:\D2VAVS and H:\MOVIE\VIDEO_TS can be found on all PCs once you mount the drive.

:helpful:

edit: names of paths

[quantum]

Good tip. Of course anyone wanting to set up an encoding farm should be expected to do some amount of network setup.

Is it otherwise working and performing as expected?

[Joergen]

Yes I've now tested it back and forth and a PC#2 encoding off PC#1 takes no CPU power from PC#1, and PC#2 is able to do about -0.05 (in CCE speed) of what it does off its own HD, and PC#1 can do exactly as it normally does.

It's working really great so far! I can get about 3.20 combined speed compared to the half with either computer doing it alone, really helps since alone each PC can take about 5 hours with a movie.

I can see it now, just map H: on a few puters, and hit the RBFarm exe on the desktop! Hey presto, 6x CCE speed :)

Thanks alot quantum. I'll buy two ATX cases soon and build barebone systems just for RBFarming!

Old McDonald had a farm.. hiiiae hiiiae hey... :D

[quantum]

Good news. I'm curious what happens if you put a 3rd and a 4th PC on there. The data transfer rate of CCE is not that high, around 1mb/sec for 1x. Your 10/100 LAN can do 10 times that and your hard drive 20 times that. So maybe 4 or even more would be okay.

The theoretical limit for your 10/100 LAN would be around CCE 10x combined speed according to my rough guess.

That's pure speculation so let us know what happens :-)

[DDogg]

You guys define the word, "geek" :D

[Joergen]

Originally posted by DDogg
You guys define the word, "geek" :D

Why, thank you :o ;)

quantum: I'll try with 3 puters tomorrow. The third one being able to do about 2.5x on its own. So I can reach about 6x I think. Then I'll bring a Pentium-M 1.8GHz laptop home from work that should help me reach around 7.5x :D

If the HD starts to be too slow or other bottlenecks appear, I can see it by looking at the cpu usage on each system.

[int 21h]

@quantum: Can you explain more what your app is doing? Is it just copying the directory as you did in the beginning? Would you be willing to release the source for it? I have some pretty decent ideas on how to improve it (with complete automation on the nodes), and I have a pretty good hand in VB...

[quantum]

Originally posted by int 21h
@quantum: Can you explain more what your app is doing? Is it just copying the directory as you did in the beginning? Would you be willing to release the source for it? I have some pretty decent ideas on how to improve it (with complete automation on the nodes), and I have a pretty good hand in VB...
It analyzes the output from the prepare phase of DVD Rebuilder and figures out how many chunks will be encoded. Then each instance of the program picks the next available chunk, parses the rebuilder.ecl to pick out the appropriate ecl, and encodes it. The instance creates a flag in the d2vavs directory which indicates the chunk has been reserved for processing so the other instances don't collide with each other.

I probably won't release the code anytime soon and I plan on making a few more tweaks.

@all: For those having CCE encoding speed issues, you could try RB Farm and see if that makes a difference. You can run it on only one PC. If my code happened to make a difference I'll gladly turn it over to jdobbs, although I'm pretty sure our code is 99% the same.

[chadp1a]

I tried RB-Farm and it works perfect. My problem is on pc#1 it get cce speed of about 1.45 and on pc#2 cce speed on average is .60

Both Pc#1 and Pc#2 are identical computers in terms of processor, memory, hardware..........

When using only dvd-rb on one pc I get speeds of 2.0+

The router I'm using is very old and I notice there are tons of collisions going on. Could the collisions be a problem maybe?

[quantum]

A few things to look at:

I'm not sure how fast your LAN must be to take advantage of 2 PC's. A 10mb LAN might be fast enough, but I'm not sure. I have 100mb which translates to 10 megabytes per second. You should test your LAN speed by copying large files between PCs and measuring the speed.

Check you have UDMA enabled on all IDE controllers. You should be using less than 10% of your CPU when copying large files. Having this set in PIO mode might work when doing a single local encode but would likely cripple a multiple network encode.

Maybe a 10/100 router or hub would help.

Otherwise I don't know what to suggest. You'll have to experiment. Let us know if you figure it out.

[Joergen]

If the network has collisions it means its a 10Mbit half-duplex network that has to wait for packets to pass back and forth. Since CCE reads and writes at the same time, half-duplex will mean your get delays from only moving in either direction at one time.

I have a 100mbit full duplex switch (switch being the key word) that doesnt limit the network in such ways.

[chadp1a]

So a new router would solve the problem then right?

I have a new linsys 5port 10/100 swich/router that I haven't tried on it yet.

Thanks for the info guys!

[Joergen]

edit: nm it was an old version of eclcce on one PC that made MPV files instead of M2V :o

[Joergen]

Originally posted by chadp1a
So a new router would solve the problem then right?

I have a new linsys 5port 10/100 swich/router that I haven't tried on it yet.

Thanks for the info guys!

Dont forget to use CAT5 cables between the router and PCs, and set the network adapters to 100mbit FULL DUPLEX in windows.

[Joergen]

Observations of the effectiveness of RBFarm:

-3pass CCE job with same project.

1PCs = 334 minutes

2PCs = 232 minutes

3PCs = 125 minutes

Instead of 5 hours (as I did often earlier) I can do it in 2! :)

I noticed that shutting down (disabling) Norton Antivirus 2004 helps about 15%. That software is weird, apparently it monitors network traffic vigorously. Shutting down Zonealarm didnt give significant gain.

[chadp1a]

@all
2 passes with CCE using RB Farm here.....

1 pc - 134 minutes

2 pc's - 87 minutes

@Joergen

Followed your advice on the sluggish network problems and it worked... thanks...

getting cce speed of 1.9+ on both pc's now :)

@Quantum
Nice addon for DVD-RB. Thanks to both you and Jdobbs

[P3gasus]

I have a question about rbfarm. I didn't understand if it assigns chunks statically or dynamiccally. I mean: if I have a p3 500 and an amd 2200xp on a lan will the chunks be assigned to the computers proportionally to the encoding speed of both? Or will they be assigned half to pc1 and half to pc2? I think it should not be difficult to implement dinamical cell assignement with a client/server application using for example java rmi or simply sockets. I could try to think about something if you think it can help.

Bye

PS. This prototype I have in mind does NOT use RB for encoding but it calls eclcce for encoding directly.
Tell me if this could cause problems.
PPS. I couldn't test rbfarm by myself because currently the link is not working.

[Joergen]

Originally posted by P3gasus
I have a question about rbfarm. I didn't understand if it assigns chunks statically or dynamiccally.
PPS. I couldn't test rbfarm by myself because currently the link is not working.

The "assignment" is the simplest and most effective method. Each PC picks up the next available chunk and encodes it, then skips the ones that are done or reserved by other PCs and encodes another chunk.

And RBFarm is standalone and only needs the prepared files in d2vavs and eclcce.

This way (and I have inbalanced setups) the fastest PC did 55% of the chunks, the middle speed (that was hosthing also) about 30% and the lowly slow puter only 15% :)

I'll try to host it someplace for all.

[P3gasus]

Originally posted by Joergen
The "assignment" is the simplest and most effective method. Each PC picks up the next available chunk and encodes it, then skips the ones that are done or reserved by other PCs and encodes another chunk.

Cool :D. That's simple but indeed effective. I can't wait to try it.
Thanks.
Bye
Pegasus

[Joergen]

Here you go all. RBFarm 1.1

rbfarm.zip (http://rbfarm.is.dreaming.org/rbfarm.zip) 24KB Dont use "save as", just left-click.

Thanks again quantum for making DVD-RB almost as fast as a transcoder for me :D

edit: Updated to 1.1

[quantum]

Originally posted by Joergen
Observations of the effectiveness of RBFarm:
-3pass CCE job with same project.
1PCs = 334 minutes
2PCs = 232 minutes
3PCs = 125 minutes Decent looking numbers and confirmed by chad1a. I wonder how much difference a switch makes over a hub in this case. I know the switch is supposed to be more efficient in theory but a real world test of a 10/100 switch vs. a 10/100 hub would be interesting.

[Joergen]

I only have switches so I cant test that one. Tomorrow I can test how a slower 700mhz PC does at the end of a 10mbit line, and if using two physical drives for read (source) and write (d2vavs dir) makes a difference.

[HanSolo00]

Pretty cool... eventually you could top out at 1 pc per chapter/chunk and get complete VTS encodes down to the 10-15 minute mark.

If I was still the IT manager at my old workplace you can guarantee I'd be trying that after hours on the LAN... just imagine how many projects you could queue up and complete overnight:)

[quantum]

I think there is a maximum you can achieve with this technique that would be hit before you could have 1 PC per chunk. You're manipulating files directly off the server so you're limited by the serving machine's ability to send and receive.

My original guess was you could achieve a combined CCE 10x on a 100megabit LAN. If each PC is running at 2x, this would be 5 PCs.

However now I'm thinking this may be higher if you have full duplex switches instead of hubs.

[EddieTH]

Originally posted by DDogg
You guys define the word, "geek" :D
A geek friend of mine had a similar reaction. ;) He also thought it was pretty cool. :p

[b1ackmai1er]

No luck getting this going yet.

Keep getting the following error in CCE:

i:\dvd-rip\d2vavs\farm_chunk0001.ecl is not a ECL for Cinema Craft Encoder SP

Is this possibly caused by the .ecl extension being lower case?


These are my rbfarm paths:
--------------------------

i:\dvd-rip\d2vavs\
d:\program files\dvd-rip\eclcce\eclcce.exe

The contents of my eclcce.ini file:
-----------------------------------
[EclCCE]
CCE=D:\Program Files\Custom Technology\Cinema Craft Encoder SP\cctsp.exe

The contents of farm_chunk0001.ecl
----------------------------------

[item]
title=V01000000001001
aud_out=0
vaf_file=I:\DVD-RIP\D2VAVS\V01000000001001.vaf
aud_file=I:\DVD-RIP\D2VAVS\V01000000001001.mpa
file_focused=0
packet_size=2048
width=720
height=576
frame_rate_idx=3
cbr_brate=6000
vbr_brate_avg=6253
vbr_brate_min=0
vbr_brate_max=9000
seq_endcode=0
dvd=1
half_width=0
half_height=0
lum_level=0
aspect_ratio=3
gop_m=3
gop_nm=4
gop_hdr=12
seq_hdr=1
all_closed_gop=0
fix_gop_length=0
samples_per_sec=44100
stereo=2
brate_idx=7
crc=1
progressive=0
alternate_scan=1
intra_dc_prec=2
encode_mode=0
vmode=2
fast_mode=0
quality_prec=24
vid_file0=I:\DVD-RIP\D2VAVS\V01000000001001.m2v
vid_file1=I:\DVD-RIP\D2VAVS\V01000000001001.m2v
vid_out=1
vaf_out=1
video_type=4
timecode=0x0000000
opv_q_factor=20
opv_brate_min=0
opv_brate_max=9000
vbr_bias=25
vbr_pass=4
use_filter=0
filter_val=6
non_linear=1
top_first=0
mpeg1=0
mpeg1_cps=1

[file]
name=I:\DVD-RIP\D2VAVS\V01000000001001.avs
frame_first=0
frame_last=12562
encode_first=0
encode_last=12562

Hope someone can help.
Am very excited about this program.

regards b1m1

[Joergen]

Originally posted by quantum
My original guess was you could achieve a combined CCE 10x on a 100megabit LAN. If each PC is running at 2x, this would be 5 PCs.


You forget one thing. You can put two 100mbit lan cards into the server pc, pull out one RJ45 to two separate 100mbit switches, and divide the node PCs to two groups for each switch. This way you not only get dedicated 100mbit for each group, but the (low cost) switches are better off not having to serve as much traffic :D

But theoretically you can calculate that the max bandwidth of a DVD is 1mbit, and lets say the returning encoded video is 0.4mbit (4000kbit). So on a PC that does 2X in CCE you suck out 2mbit and return 0.8mbit, and add some room for overhead bandwidth you can say its 3mbit per PC. A 100mbit network can, in reality, do about 50mbit per direction on low cost adapters. That leaves room for 16 PC's of 2X caliber.

Today I'm testing how much encoding on the serving PC hurts the network performance. As CCE seems to run like a High Priority program (everything else stands still for seconds) eventhough it is in normal priority.

[quantum]

@b1ackmai1er
I believe this happens when you run the prepare phase of DVD RB with something other than options->cce options->cce sp (2.66+)

For instance if you have cce basic selected you'll get this error. You should also get this error if you try to drag and drop rebuilder.ecl into CCE.

Also make sure you have the latest eclcce.

[quantum]

Originally posted by Joergen
But theoretically you can calculate that the max bandwidth of a DVD is 1mbit, and lets say the returning encoded video is 0.4mbit (4000kbit). So on a PC that does 2X in CCE you suck out 2mbit and I don't know if theory works out in practice. I based my figures on my network bandwidth meter. It shows me pulling around 10megabit from the server when encoding at 1x. This is about 10% capacity of my 10/100 LAN. The return stream is much less, around .5megabit.

[Joergen]

Results:

I used an XP2100+ PC with WinXP and 1GB of DDR333 RAM, and two physical 7200rpm drives to serve the source and d2vavs. Then an AMD64 3200+ to encode off that PC. When the server is not encoding the AMD64 encodes at 100% CPU usage, but when the server is also encoding the AMD64 usage graph fluctuates from the low 40% to around 90% constantly.

This leads me to think it would be best to serve on the slowest (400-900mhz) PC on the network with a decent HD, or if available, dedicate a faster PC just for serving. Dedicated serving can get the final edge out of RBFarm.

quantum: is there a low priority switch for CCE or ECLCCE? That might help the serving PC keep up with the underlying tasks.

[b1ackmai1er]

Hi Quantum,

Thanks for such a quick reply.

Yes I am using CCE 2.50 and have set Rebuilder to use it using ECLCCE 1.8b. This combination works fine for rebuilder.

I am confused. Are you saying that I cannot use this version with RBfarm?

My understanding was that the whole purpose of the ECLCCE was to make versions such as 2.50 work with ECL files, otherwise there would be no need to specify the ECLCCE path and we would be running CCE directly with each farm.ecl file.

Are you able to clarify that for me?

Thanks again.

Regards b1m1.

[quantum]

Originally posted by Joergen
quantum: is there a low priority switch for CCE or ECLCCE? That might help the serving PC keep up with the underlying tasks. Yes, run eclcce (which will launch CCE),

help->about->eclcce tab
CCE Process Priority->idle

You can also change it manually in the eclcce.ini file.

[Joergen]

Originally posted by quantum
Yes, run eclcce (which will launch CCE),

help->about->eclcce tab
CCE Process Priority->idle

You can also change it manually in the eclcce.ini file.

Excellent! I never noticed that tab (RTFM!) ;)

Well that seems to do it. The server is still doing 100% and the AMD64 is doing 100% as before. Now the 3-8% of CPU usage the server had from the others using its resources gets done FIRST and not last.

This might be a semi-important note for optimizing your RBFarm people
:helpful:

[b1ackmai1er]

Yes Quantum you are right!

I had RBfarm set to Rejig for preparation pass.

Will try again.

Damn I promised my wife I would be in bed before midnight tonight :(

Thanks b1m1

[quantum]

I don't think it will work with 2.50 yet, but I think I know what the problem is.

I'll post an update soon.

[quantum]

I updated the link on page 1 to version 1.1. This should allow CCE SP 2.50 to work with RB Farm. The way I was parsing the rebuilder.ecl file worked with CCE 2.66+ but not with 2.50. It should work with both versions now.

I still don't think it will work with CCE Basic, but I don't know for sure.

[Infiniterb]

For some reason this seems like it's moving insanely fast for my encodes.

I've got a 3 pass CCE job going, but the process to encode doesn't seem like it's doing anything. It just does the first pass extremely fast, then the second, then third.

I did the prepare process on one PC, and have that drive mapped to Z:

RB's paths are the folllowing:
The source is Z:\JurassicPark
The Destination is Z:DVDWRKDIR

For DVD RB Farm:

On PC ONE

D2VAVS Path: Z:DVDWRKDIR\D2VAVS
ECLCCE path: C:\Program Files\ECLCCE\eclcce.exe

On PC TWO

D2VAVS Path: Z:DVDWRKDIR\D2VAVS
ECLCCE path: C:\Program Files\ECLCCE\eclcce.exe

Is there anything else I'm missing?

[Joergen]

Originally posted by Infiniterb

The source is Z:\JurassicPark
The Destination is Z: DVDWRKDIR



You forgot to add a \ after Z:\DVDWRKDIR?

edit: and make sure you have latest avisynth installed and the mpeg2decdg.dll copied to its plugins dir (or the same dir specified insider dvd-rb if you used it).

[quantum]

RB Farm troubleshooting 101:

- Ensure you can encode correctly with DVD Rebuilder on the main PC.
- Ensure you can encode correctly with RB Farm on the main PC.
- Start, then abort an encode using DVB Rebuilder on the main PC. On secondary PCs, ensure you can open item.ecl and encode using eclcce.

If you can do these things, you should have no problems.

[b1ackmai1er]

Hi Quantum,

I am just about to download 1.1 and give that a try.

What I did find this morning is that if I try to load a farm.ecl file manually using CCE started with ECLCCE then I would get the same error as described previously.

However if I modified the farm.ecl file by putting the comments (;;) from REBUILDER.ECL at the top of file then it would load without error.

Regards b1m1

[b1ackmai1er]

Hi Quantum,

Just confirming that RB farm 1.1 works fine with CCE 2.50 (+ECLCCE1.81).

This is a dream come true.

Thanks b1m1

[acido]

@quantum
Thanks a lot for this tool, it worked really good for me cutting encoding time by half (2 pc used).
Hope it will be integrated in dvd-rb !

[Alex Z]

Originally posted by quantum
I completed a very rudimentary prototype. It works but it's too basic to post and I expect problems so I'd like to get an email correspondence going with testers to avoid cluttering up jdobbs forum.

@EddieTH I sent you a copy via email.
@Joergen I sent you a PM with the location and details.

Anyone else interested in testing this PM me (preferrably with an email address) and I'll send you a copy.

@quantum
Can psexec.exe (or simular) be used in future version of RB farm?
So you can start and control remote process remotely from one PC running RB farm app.

az

[EddieTH]

@quantum
Is there any chance for a version with CCE Basic support?

[Joergen]

I would recommend a program called RAdmin. It loads at startup, is very fast and low-cpu usage and you can control the other PC fully.
Far superior to software like PCAnywhere or WinVNC.

[Alex Z]

Originally posted by Joergen
I would recommend a program called RAdmin. It loads at startup, is very fast and low-cpu usage and you can control the other PC fully.
Far superior to software like PCAnywhere or WinVNC.


psexec is not your typical remote program. Its purpose is to execute a task (process) on remote PC (interactively or not). To make it even more simple to understand you can start calc.exe on PC 'B' using PC 'A'. In this scenario, RB farm will send command to execute ecllcce process on \\xyz pc wait for it to finish, execute next.

az

[quantum]

Originally posted by EddieTH
Is there any chance for a version with CCE Basic support? I would add support, but I only have the CCE Basic trial version which apparently has a different ECL format than the full version, so I can't do any testing.

[gunga]

Hello Quantum,

It seem everytime I click encode in RBFarm 1.1, I get this error "GetFileAttributes()Failed" on my networked computers. It works fine on the source computer, but not on any networked. I can map to any machine on the network and see all folders. I'm using CCE 2.67.00.23/eclcce 1.8b and DVDRebuild 0.44 on all machines. Any ideas as to why this is happening?

Thanks...

[EddieTH]

Originally posted by quantum
I would add support, but I only have the CCE Basic trial version which apparently has a different ECL format than the full version, so I can't do any testing.
If you decide to try it, let me know. I have the full version to test with.

[quantum]

I updated the link on page 1 to version 1.2. This version might work with CCE Basic, but it's untested. It definitely does not work with CCE Basic Trial which does not have ECL support.

If you want to try CCE Basic, you must type the filename manually on the "Path to ECLCCE" line in RB Farm.

Please let me know if it works.

[EddieTH]

@quantum
I'm in the middle of an encode right now, but I'll test it on my next project (later tonight) and let you know. Thanks.

[quantum]

Updated to version 1.3.

Added a new feature "Run Automatically"

The "Run Automatically" check box will cause the current instance of RB Farm to periodically detect if an encode has been started in the d2vavs directory. If a job has been started, RB Farm will automatically join in and process the next available chunk. You must start an encode on one PC by pressing the "Encode" button before any other instance of RB Farm will begin encoding automatically.

With this you could leave RB Farm running on PC's designated for encoding and not have to manually start the encode process on each PC.

[Joergen]

Originally posted by quantum
Updated to version 1.3.

Added a new feature "Run Automatically"

Ooh :eek: A true hard-core RBFarming option :D

[b1ackmai1er]

I am less than thrilled with the performance of my 2 PC farm.

HOST - CELERON 950, 100Mhz BUS, 768 Meg RAM = CCE SPEED .550
FARM - CELERON 600, 66Mhz BUS, 384 Meg RAM = CCE SPEED .150

I was wondering if others with low end machines could post some
speeds for comparison.

RB Farm is such a great tool but I guess I might just have to upgrade after all.

Also has anyone noticed that CCE encoding dies when a shared directory (other than those in use) is unshared?

Regards b1m1

[Joergen]

Originally posted by b1ackmai1er
I am less than thrilled with the performance of my 2 PC farm.


Definetly something wrong with your PC#2 or your PC#1 needs to put CCE on IDLE USAGE through ECLCCE.

My third PC is a Celeron2 850mhz 100fsb (oced) and does 0.46 when an XP2100+ does 1.20 and A64 3200+ does 2.10 (on the same project).

[EddieTH]

I can verify that RBFarm works with CCE Basic. I have 2 machines encoding with it right now.

Edit: I should mention that I'm starting RBFarm at Idle priority so it will start CCE at that priority as well. I put this batch file in the RBFarm directory (OldMacdonald.bat):

start /low RBFarm.exe


@Quantum: Thanks for the help getting it set up right. It's been so long since I've used any of my "serious" computer skills I've started forgetting the basics.

[puddy]

Anybody else getting oversized output with RB Farm?

If I encode on a singer computer using only
DVD-RB I get a "perfect" 4.32 GB disc.

Using RB Farm I got a 5.00 GB disc. All settings
were the same. Farm consisted of 3 computers (two
P4's and a P3). The RB Farm encode appears to be
fine in all other aspects, it is just too large!

Here's my stats:

Disc: Brother Bear Disc 2 NTSC
DVD Decrypter v3.2.1.0: Ripped in File Mode

DVD-RB v.45
- Removed: English DTS and AC3 French using DVD-RB
- Dynamically Assign Cell Bitrates Enabled
- Half-D1 and Half Space for Extra Enabled
- ConvertToYUY2() Enabled
- AudioDub(BlankClip()) Enabled
- CCE SP (+2.66+) Mode
- VBR_Bias=25
- Quality_prec=16
- VBR_Passes=3
- Add MPEG2DEC3DG to AVS File Enabled

RB Farm v1.3
CCE SP v2.67.00.23
eclcce v1.81
AviSynth v2.54 (all other versions uninstalled & registry cleaned)
MPEG2Dec3dg.dll v1.0.1.0 (put in AviSynth plugins dir)

Rb Farm is just very, very cool. Hope to be using it a lot in the future.

Ideas? :)

puddy

[quantum]

Before you ran the "rebuild" phase after the encode did you remember to deselect audio streams? Your deselections are not retained between starts and stops unless you save/load a project.

[wgw]

Originally posted by Joergen
Definetly something wrong with your PC#2 or your PC#1 needs to put CCE on IDLE USAGE through ECLCCE.

My third PC is a Celeron2 850mhz 100fsb (oced) and does 0.46 when an XP2100+ does 1.20 and A64 3200+ does 2.10 (on the same project).

b1ackmai1er, I'd say Joergen has the right idea. Set CCE on PC#1 to low priority. My dual MP2000+ does 2.2 while my second PC with a 1ghz Athlon does 0.9. However, if I unrar an archive on PC#2, CCE nearly stops processing completely even when both CCE and Winrar are both set to low priority. PC#1 will also drop to 0.45 if I am doing a lot of processing while CCE is running in the backround. CCE seems to be one app that could really use dual CPU's unless you can leave your computers alone for 6 hours or more while it is processing.

[puddy]

quantum: That is probably it! I'll test now.

puddy

[puddy]

That was it... I just had to go back and
uncheck the audio tracks and rebuild.

Thanks so much. :)

puddy

[quantum]

Care to share your numbers with us? The time to encode on one PC and the time for 3, and the processors?

[puddy]

1. P4 2.8 GHz, 800 MHz Bus, 200 MHz FSB, 512 MBs DDR-SDRAM PC3200
2. P4 2.66 GHz, 533 MHz Bus, 133 MHz FSB, 1 GB DDR-SDRAM PC2700
3. P3 600 MHz, 100 Mhz Bus, 100 MHz FBS (Laptop), 288 MBs 100MHz SDRAM

- No overclocking on any machine
- Linksys 100 Mbit switch with Cat-5e & Cat-6 wires

On the Brother Bear Disc 2 DVD (see previous post) the last CPU (the P3) finished up after 106 mins and only did 5 of the 76 cells. Plus 6 mins Prepare time and about 10 mins Rebuild time. So, a total of about two hours.

Machine #1 did the same disc in a little over 4 hours (240 min) standalone using one-click mode.

In both cases the source and destination were on different physical drives. But in the RB Farm run the source and destination were through mapped drives, even on the master CPU.

I noticed that when I ran the standalone test the Prepare process for this DVD was about 4 min and it was about double that through the mapped drives in preperation for the Rb Farm run. So, there is overhead that wouldn't normally need to be there. Maybe that is something could be looked at more?

So, that is just about what you'd expect from a setup like this, which is really cool.

I bet you'd be able to track some stats automatiaclly with a little coding that we could cut & paste back here for reference. That would be great! :)

Thanks again!

puddy

[puddy]

RB Farm Stats:
Disc: Dickie Roberts 16x9 NTSC (7.27 GBs Originally)
Removed: English 2-CH AC3 and French AC3 using DVD-RB
Prepare phase: 9 min
75 chunks total, VBR_passes=6
Machine #1: Processed 40 chunks in 235 mins
Machine #2: Processed 31 chunks in 228 mins
Machine #3: Processed 4 chunks in 229 mins

DVD-RB failed upon rebuild at segment 8 VOBID:6 CELLID:1 with the Run-time error '9': Subscript out of range error.
This same error happens with DVD-RB standalone in one-click mode as well. :(

My puddyFarm Stats:

HARDWARE
1. P4 2.8 GHz w/HT, 800 MHz Bus, 200 MHz FSB, 512 MBs DDR-SDRAM PC3200
2. P4 2.66 GHz, 533 MHz Bus, 133 MHz FSB, 1 GB DDR-SDRAM PC2700
3. P3 600 MHz, 100 Mhz Bus, 100 MHz FBS (Laptop), 288 MBs 100MHz SDRAM

No overclocking on any machine
Linksys WRT54G 100 Mbit switch over Cat-5e & Cat-6 wire (not using WI-FI in the puddyFarm)

SOFTWARE
DVD Decrypter v3.2.1.0: Rip in File Mode
DVD-RB v.45
- Dynamically Assign Cell Bitrates - ENABLED
- Half-D1 and Half Space for Extra - ENABLED
- ConvertToYUY2() - ENABLED
- AudioDub(BlankClip()) - ENABLED
- Add MPEG2DEC3DG to AVS File - ENABLED
- Run Encoder(s) Minimized - ENABLED
- CCE SP (v2.66+) Mode
- VBR_Bias=25
- Quality_prec=16
- VBR_Passes=6

RB Farm v1.3
eclcce v1.81
CCE SP v2.67.00.23
- Remove Cancel Button - ENABLED
- Priority set to IDLE

AviSynth v2.54 (all other versions uninstalled & registry cleaned)
MPEG2Dec3dg.dll v1.0.1.0 (put in AviSynth plugins dir)

Source and destination were on different physical drives on machine #1.
All machines, including the main, used two separate mapped drives for source and destination.


puddy

[EddieTH]

Since I tweaked some memory settings on my laptop (to compensate for low memory) and stopped using the onboard NIC on my main PC I've been getting between 1.5 and 1.9 on my main PC (2.4GHz P4/768MB), half that on my laptop (Celeron 1.7GHz/128 MB), and a third that on my wife's PC (Athlon 1GHz/256MB).

I'm getting the wife a new PC for mother's day, so I'll have another fast PC to use :cool: (when my wife's at work anyway). I'm considering getting a good NIC for the PC she's using now and just using it to serve the files.

[b1ackmai1er]

@wgw

Thanks for posting your celeron 850 stats. This confirms for me that my HOST PC is running fine. My FARM pc is running better now at around .250 . I think the movie I was encoding as a test (Lizzie McGuire) was slipping me a red herring, encoding speed was very low at the start of the movie but picked up speed later on. I suspect the animated character at the start of the movie was causing avisynth to load the decomb filter but I don't know how to confirm that.

@eddieTH

Wow! You run CCE with 128Meg of RAM! That's extreme!

@Quantum

Can I suggest you modify the "Run automatically" feature to do it's check every minute rather than every second.
Also can I suggest adding a "minimize on startup" so I can put it in my startup folder.


@puddy

Do you remember what you Machine 3 P3 600 CCE speed was?

@Joergen

Thanks for you suggestion. I tried various priorities on both my host and farm PC. Basically I found that High priority on the HOST killed the farm machine. Hight Priority on the farm machine made little difference but made the machine unusable for use. Low priority on both machines slowed both down a little. I found normal priority on both machines was about optimum.

[quantum]

Originally posted by b1ackmai1er
Can I suggest you modify the "Run automatically" feature to do it's check every minute rather than every second.It should be checking every 20 seconds. Too quick?
Also can I suggest adding a "minimize on startup" so I can put it in my startup folder.Put a shortcut in your startup.
In shortcut properties, change run->normal window to run->minimized

[puddy]

Originally posted by b1ackmai1er
Do you remember what you Machine 3 P3 600 CCE speed was?Something like .41 I think on the P3 600.

Would be cool of RB Farm would create a small log file of these stats that we could cut and paste into the forum for comparisons. :)

puddy

[puddy]

Originally posted by quantum
Put a shortcut in your startup.
In shortcut properties, change run->normal window to run->minimized Minimize to tray would be awesome. :)

[EddieTH]

Originally posted by b1ackmai1er

@eddieTH

Wow! You run CCE with 128Meg of RAM! That's extreme!


I found a registry hack that gave CCE a major speed boost. I started out with a speed of .30 to .35, but after the registry hack I get .70 to .95. Once I get my income tax refund (within a week) I'm going to be using the laptop as a laptop again (needs a new LCD) and I'll have a better PC to replace it with.

[quantum]

Can we have more details on the registry hack?

[EddieTH]

Oh yeah, forgot to include the details. :confused:

Look for the registry key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\PrefetchParameters\EnablePrefetcher

Set the value to 0 (Disable). That disables prefetch. CCE Starts a little slower but it performs a lot better. As with anything, YMMV.

Edit: I found this (don't remember where) listed only as a Windows XP hack, but that's the only Windows OS I'm running, so I haven't tried it on anything else.

[quantum]

Originally posted by Joergen
I would recommend a program called RAdmin. It loads at startup, is very fast and low-cpu usage and you can control the other PC fully.
Far superior to software like PCAnywhere or WinVNC. I use RAdmin and it works well. I just started using Remote Desktop which is part of XP. For some reason I never tried it before. I was very surprised at the great performance. Great combination of low CPU and high speed. Better than RAdmin IMO. It requires XP for the server part, but apparently the client part can run on other versions of windows.

[EddieTH]

I use Remote Desktop to run all my PCs for encoding and it works great. It's really just a limited version of Terminal Services. It's also nice for keeping your computer busy while you're at work. Of course I work from home so that's not really an issue for me.

[b1ackmai1er]

Hi Quantum,

Thanks for your feedback. I thought I could do something like that by creating a .PIF file or shortcut. But can't test it till my encode has finished.

How about an option to start encoding on startup? Or will "run automatically" do that even if the encoding process hasn't been started yet.

The reason I mentioned the checking time period was because I was looking at the network bandwith and saw packets heading out every second so I presumed RbFarm was doing a check every second. That made me think about the disk thrashing that would occur as every farm machine checked every second. Maybe the data is cached and it doesn't make any difference. Possibly I was mistaken and another application was accessing the network.

For one click integration with rebuilder is in not possible to set the Rebuilder CCE Application path to point to RBFarm or will rebuilder continue on to final step after the first RBFarm CCE encode is finished? If so maybe RB could add your completion check into Rebuilder. Is that as easy as it sounds?

I can't sleep. This is exciting stuff. Now to try that reg patch!

Regards b1m1

[SAPSTAR]

Originally posted by EddieTH
Oh yeah, forgot to include the details. :confused:

Look for the registry key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\PrefetchParameters\EnablePrefetcher

Set the value to 0 (Disable). That disables prefetch. CCE Starts a little slower but it performs a lot better. As with anything, YMMV.

Edit: I found this (don't remember where) listed only as a Windows XP hack, but that's the only Windows OS I'm running, so I haven't tried it on anything else.

Well...I tried on two boxes, a PIII 1000Mhz and a Celeron 1.7Ghz...NO change ?!?!? Maybe on the slowest computers it's more efficient....

[quantum]

I didn't see any change either on my Athlon 1.4.

[b1ackmai1er]

You could try this utility:

http://www.jester2k.pwp.blueyonder.co.uk/downloads/Windows-XP-Prefetch-Clean-And-Control.exe

Regard b1m1

[EddieTH]

Originally posted by SAPSTAR
Well...I tried on two boxes, a PIII 1000Mhz and a Celeron 1.7Ghz...NO change ?!?!? Maybe on the slowest computers it's more efficient....
I couldn't remember the specifics of why I did this, so I did some more research. It's specifically for low memory machines, so the speed of the CPU isn't nearly as important here. I only used it on the laptop because the others have decent memory.

[puddy]

RB Farm Stats for this run:
Disc: Italian Job NTSC (7.64 GBs originally)
Removed: English 2ch AC3 and French 6ch AC3 using DVD-RB
Prepare phase: 11 min
136 chunks total, VBR_passes=6
Machine #1: Processed 80 chunks in 239 mins (Server)
Machine #2: Processed 56 chunks in 245 mins (Remote)

DVD-RB failed upon rebuild at segment 2 VOBID:3 CELLID:1 with the "Run-time error '9': Subscript out of range" error. This same error happens with DVD-RB standalone in one-click mode as well. :(

My puddyFarm Stats for this run:

HARDWARE
1. P4 2.8 GHz w/HT, 800 MHz Bus, 200 MHz FSB, 512 MBs DDR-SDRAM PC3200 (Server)
2. P4 2.66 GHz, 533 MHz Bus, 133 MHz FSB, 1 GB DDR-SDRAM PC2700 (Remote)

No overclocking on any machine
Linksys WRT54G 100 Mbit switch over Cat-5e & Cat-6 wire (not using WI-FI in the puddyFarm)

SOFTWARE
DVD Decrypter v3.2.1.0: Rip in File Mode
DVD-RB v.45
- Dynamically Assign Cell Bitrates - ENABLED
- Half-D1 and Half Space for Extra - ENABLED
- ConvertToYUY2() - ENABLED
- AudioDub(BlankClip()) - ENABLED
- Add MPEG2DEC3DG to AVS File - ENABLED
- Run Encoder(s) Minimized - ENABLED
- CCE SP (v2.66+) Mode
- VBR_Bias=25
- Quality_prec=16
- VBR_Passes=6

RB Farm v1.3
eclcce v1.81
CCE SP v2.67.00.23
- Remove Cancel Button - ENABLED
- Priority set to IDLE

AviSynth v2.54 (all other versions uninstalled & registry cleaned)
MPEG2Dec3dg.dll v1.0.1.0 (put in AviSynth plugins dir)

Source and destination were on different physical drives on machine #1.
All machines, including the main, used two separate mapped drives for source and destination.

puddy

[Joergen]

It's the half-d1 option that causes the error 9 for alot of people now I think.

[puddy]

Originally posted by Joergen
It's the half-d1 option that causes the error 9 for alot of people now I think. I think you are right, but I like to point it out so that jdobbs can have the largest base of info to work with in tracking down the bugs. He's doing a great job already...

puddy

[quantum]

Updated the program to 1.4 and added a feature (from the readme)

REBUILD AUTOMATICALLY
The "Rebuild Automatically" check box will cause the current instance of RB Farm to attempt to rebuild the project after encoding using DVD Rebuilder. You should only select this option on the PC where DVD Rebuilder is installed and where you ran the prepare phase. DVD Rebuilder should be running with the settings for the current project correctly chosen, i.e., audio streams deselected, etc. The mode should not be one click mode, and "suppress warning prompts" should be selected.

I expect monster bugs with this. It's never easy remote-controlling another program. But if it works, this should further reduce the number of steps necessary to run a farm.

Theoretically, this could be the workflow:
1 - Run prepare with DVD Rebuilder
2 - Press encode with RB Farm (other PC's join in automatically, rebuild is done automatically)
Finished

[DDogg]

I'm refreshing like mad but the main link is still 1.2 and the alternate link is 1.3 for me. Suggestions? (yeah you guys are geeks, but damn smart geeks :) ) /Edit-Add/ Disregard, it came up. Nice job. I particularly appreciate the new lazy geek features. Heck, don't even have to get out to the chair now. (not tested yet)

[quantum]

Originally posted by DDogg
I particularly appreciate the new lazy geek features. Heck, don't even have to get out to the chair now. I guess I should finally add the ability to automatically start encoding without having to press the encode button, which I could probably do. Then it would be back to 1 click, prepare, and that's it :)

[EddieTH]

An update on my setup and what kind of speed I get from it. I found somebody to get another 128MB RAM from for my laptop, so it now has 256MB and I changed the prefetch registry setting to 1 (prefetch only for applications). My speed with that PC has gone up about 20%.

Source: Local hard drive in firewire external enclosure
Drive lettter Z: (mapped to Z: on remote PCs)

Working: Folder from internal hard drive mapped to W: on all PCs

*PC1: 1.50-2.15 P4 2.4GHz / 768MB RAM / Linksys NIC
PC2: 0.85-1.15 (laptop) Celeron1.7GHZ / 256MB RAM / Onboard 3Com 920 NIC
PC3: 0.50-0.85 Athlon 1GHz / 256MB (SD)RAM / Linksys NIC
*When encoding by itself, my main PC runs at 1.7-2.3.

Switch: Actiontec POS (okay technically it's an Actiontec 1524) DSL Router/Switch

Right now my biggest problem is having them all connected through a poor switch. I've never had a reason to use the network this way in the past, so it was never an issue, but since I started experimenting with networked encoding I've decided to upgrade soon.

[Mutant_Fruit]

I'm not sure if this has been mentioned already or not, but here's why bandwidth usage is so high for the outgoing connections from the server to the clients.

As far as i know, AVISynth does what it says on the tin, its synthesis an AVI file. It does this by on-the-fly decoding of the mpeg into an uncompressed format, which it then feeds to the different applications who open the .avs file. Thats why bandwidth usage is so high.

[Joergen]

Oh yes, good point. Also I wonder if the new mpeg2dec version with improved random access helps in this.

edit: no wait, avisynth runs on the reading PC and the conversion from mpeg2dec->avisynth avi is done in the RAM of the reading PC, not the server.

[EddieTH]

I just finished Matrix Revolutions (movie only from DVD Shrink) in 87 minutes encoding time and 19 minutes preparation and rebuilding. I noticed that CCE's speed on my main PC kept going down during the course of the encode process. When the other 2 machines stopped encoding their last segments and it got to encode alone again it shot from 1.29 to 2.11. I'll have to investigate this further, but I'm still happy with the time.

@quantum: I had the Rebuild Automatically option checked, and nothing happened. I also had DVD-RB running with all the correct settings.

[quantum]

Originally posted by EddieTH
@quantum: I had the Rebuild Automatically option checked, and nothing happened. I also had DVD-RB running with all the correct settings. Did you have the path to DVD Rebuilder entered in RB Farm? Did you have rebuild checked on the same PC you pressed the encode button? The first thing that should have happened is it should have shut down DVD Rebuilder automatically before it started encoding. If that didn't happen, you know it's not working.

It should also create a file farmproject.rbd in the rebuilder directory which holds the current settings.

[EddieTH]

Originally posted by quantum
Did you have the path to DVD Rebuilder entered in RB Farm?
Yes
Did you have rebuild checked on the same PC you pressed the encode button?
Yes
The first thing that should have happened is it should have shut down DVD Rebuilder automatically before it started encoding. If that didn't happen, you know it's not working.
Okay, I can say for sure that DVD-RB didn't close, but I just started another RBFarm encode with Auto Rebuild enabled and this time it did shut down DVD-RB. I'll let you know for sure in the morning.

[EddieTH]

@quantum: I didn't get my movie started last night due to network problems. This morning I got everything working again and tried the Automatic Rebuild option again. When I hit encode, nothing happened for a couple of seconds, then a new DVD-RB window opened (I still had the window open from the Prepare step) and I got this error:
Couldn't find DVD Rebuilder window at rebuild time.

Edit: Never mind. I shouldn't do this before I've had enough coffee. I forgot to change the working directory in RBFarm on my main PC so it was trying to rebuild a movie I finished yesterday. When I fixed the directory issue it seemed to work. Unfortunately, as I'm typing this, my network is #@!*# up again. I can't wait to get a new switch (next week hopefully).

[quantum]

Actually there were a few problems and it probably wouldn't have worked it most cases.

I fixed the major problems and it should be more robust now. You can test it this way:

If the d2vavs\farm_status directory, ensure farm_finished and farm_rebuilt are deleted.

Ensure farm_chunk_startedXXXX (choose the last number) is deleted.

Then start RB Farm and with the "rebuild automatically" option selected, press Encode.

[Joergen]

Btw since DVD-RB doesnt remember the audio you unticked, doesnt automatic rebuild mess that up?

[quantum]

Good question. The first thing that happens when you have auto rebuild enabled is RB Farm sends keystrokes to DVD Rebuilder to save the current settings (including audio deselections) in farmproject.rbd. Then when the encode is complete, DVD Rebuilder is launched and farmproject.rbd is reloaded, and the rebuild is initiated.

[Joergen]

:confused: :eek: :cool:

Wow! But be careful not to make RB Farm auto-migrating, or it will take over an entire network of computers, become self-aware and launch the nukes at 18:18.

[Joergen]

Hmm I just made an observation:

When rebuilding with mapped drives (O: and P: ) the rebuild time was 42 minutes.

When rebuilding with real drives (D: and E: ) the rebuild time was 21 minutes.

Under WinXP 1GB RAM and DMA enabled etc.

:helpful:

[quantum]

Prepare and rebuild are disk intensive. The best performance should be achieved by doing these on the serving computer where the files are local.

[Joergen]

Yes thats what I was saying (I can see I couldhave explained more).

It was the main computer but I rebuilt with the mapped paths (so D: was mapped to O: and E: to P: ). I can understand microsoft didnt intend somebody to use the network mapping feature for cloning their discs but clearly the mapping makes the drive access times twice as long.

And D: is HD#1 (7200rpm maxtor) and E: is HD#2 (another 7200rpm maxtor), both NTFS4 (or whatever version winxp likes to do it with).

Feel free to try it on your own. :)

[quantum]

Now I understand. Maybe you can change the real drive letters to O and P using disk manager and map drives D and E if you still need them for your other stuff. Or maybe there is a more efficient way to clone a drive letter.

[Joergen]

I dont want to reverse the mapping cause then my games would load slower when I play ;)

But its no biggie to change the paths before rebuilding, just a few clicks.

But if somebody else is wondering why the rebuild takes longer than usual.

[quantum]

Try the subst command. At the cmd prompt, type:

subst o: d:\

o being the new drive and d:\ being the path

[Joergen]

OMG I havent seen that command for years! Thanks for reminding me.
I'm rebuilding now with substituted paths for both drives and the speed seems consistant to the 21minute rebuild with real paths.

edit: Yep, subst paths gave me 22minutes. :)
I'll just add them to a batch file and copy it to the startup folder to map them after each reboot.

:helpful:

[quantum]

New update 1.6

Upgraded auto encode functionality. You no longer need to press the encode button on a single PC to start the encode process. This should allow one click functionality on a farm. You should be able to press "prepare" in DVD Rebuilder and RB Farm will automatically do the rest.

There isn't much more that can be done with this. The only thing that isn't being supported is batch jobs. I haven't heard any requests for that, so hopefully this will be the last update for a while, except for bug fixes.

[EddieTH]

I hadn't thought of subst. I've just been using a local Z drive that's mapped on the other PCs for my source VOBs and a mapped drive on all PCs for the working directory. Then I've been changing the working directory in DVD-RB to the local drive instead of the mapped drive before I rebuild and it runs at full speed that way. Using subst means I can be even lazier. Very cool.

@quantum: I agree. There isn't much more to be done with this. It does everything it was intended to do already. Thanks for a nice program.:)

[b1ackmai1er]

Would some kind soul be able to mirror the latest version of RBFarm for me? I have been unable to download any of the last three versions even after clearing my browser cache.

Thanks b1m1

[b1ackmai1er]

@Joergen

Did you try out the subst command? I found I couldn't share a subst'ed drive and was wondering what your milage was.

@quantum

Any chance you could program RBFarm to accept network paths instead of drive/folder paths. i.e. \\celeron950machine\d:\DVD-RIP. This would also require the folder paths in the ecl files to be updated before executing eclcce. ??

Also like to suggest Rb Farm start encoding the largest segments first so that we don't end up with a situation where encoding has finished except for the largest segment which encoding on the slowest machine.

Regards b1m1

[Joergen]

You cant use the subst on network drives, only local ones. But that was my purpose, to use it to create virtual drive letters for the main computer so I can run the prepare phase with the same paths as all the nework computers. But the subst way of drive mapping is way faster on the localhost level than the network neighborhood way.

edit: Yeah that largest segment first (or at least not last) is a good idea. That way if the worst segment is picked up by the slowest PC it has time to finish it.

For now I've usually hit END and cancel on the slowest PC if it picked up a huge segment just as the faster pc's are finished, and then just encode that last bit with a faster pc.

[smlong426]

I can encode just fine with RB-FARM, but when it finishes, I load Rebuilder and click 'Rebuild' I get an error saying "There is nothing to do!"

Any suggestions?

[Joergen]

Well maybe one of the computers isnt writing to the same D2AVS path? If theres even one .m2v missing DVD-RB will say theres nothing to do.

Also an old version of ECLCCE writes .MPV files instead of .M2V, so update to 1.8 or newer on all PCs.

[smlong426]

Originally posted by Joergen
Well maybe one of the computers isnt writing to the same D2AVS path? If theres even one .m2v missing DVD-RB will say theres nothing to do.

Also an old version of ECLCCE writes .MPV files instead of .M2V, so update to 1.8 or newer on all PCs.

Updating EclCCE made this problem go away. Thanks.

[quantum]

Originally posted by b1ackmai1er
Did you try out the subst command? I found I couldn't share a subst'ed drive and was wondering what your milage was.
You don't need to share the subst drive, you share the original drive. On the other computer, you assign the correct drive letter when you map. This assumes the drive letter where the dvd files are stored is taken on other computers. Otherwise this is a non-issue:

computer1:
- dvd stored on drive d:
- subst m: d:\
- share d: as moviedrive

computer2:
- drive d: is taken
- map \\computer1\moviedrive as m:

The path for DVD Rebuilder and RB Farm(s) would be m:\yourdirhere

[b1ackmai1er]

Thanks everyone.

[NobbyNobbs]

Originally posted by quantum
There isn't much more that can be done with this. The only thing that isn't being supported is batch jobs. I haven't heard any requests for that, so hopefully this will be the last update for a while, except for bug fixes.

I would like the possibility to use RBFarm for batch jobs, if I go to work leaving 3 computers with one DVD to do each, when I get home one will not be finished, one will just have finished a couple of hours ago, and the fastest will have finished several hours ago.
If I could use RBFarm for this, I could have finished 3 to 4 DVD´s.

I often use a long time with different tools (DVD Shrink, DVD ReMake, IfoEdit, etc.-) to prepare a disc for encoding, so it would be nice to have the computers finish of the work when I´m sleeping, or at work. :)

[wgw]

Originally posted by b1ackmai1er
Would some kind soul be able to mirror the latest version of RBFarm for me? I have been unable to download any of the last three versions even after clearing my browser cache.

Thanks b1m1

This happened to me once. Try manually deleting all temporary internet files in your cache folder including cookies.

[b1ackmai1er]

@Quantum

RbFarm is working really well now. I would appreciate batch jobs support as well.

Thanks for such a great program.

Regards b1m1

[b1ackmai1er]

@Quantum

I must have had a hiccup with my RB Farm encode today because I had to restart it. It finished ok but when I Rebuilt with Rebuilder I got an error. I went back into the rbfarm directory and found that one segment had not finished encoding.

There was a farm_chunk_started0035 but no farm_chunk_finished0035 file. However there was a farm_finished file.

I restarted RBFarm and expected it to pick up this incomplete farm chunk and restart it, but it didn't.

I deleted the farm_finished file and restarted again but it still didn't pick it up.

Finally I succeeded by deleting both the farm_finished file and the farm_chunk_finished0035 file.

I thought about this for a while and then realized that this makes sense because RbFarm has no way to know if a farm PC has crashed or is still running that incomplete chunk. But it does know there are no more unstarted chunks so it writes a farm_finished file

So I wanted to suggest that you add a message that notifies the user in such a scenario. ie when you start or shut RB farm it checks if there is a farm_finished file. If there isn't it continues on respectively. If there is it should then check if it has any farm_chunk_startedXXXX files that are missing corresponding farm_chunk_finishedXXXX files and alert the user.

Hope this is clear and not to pedantic.

regards b1m1

[quantum]

If an encode is improperly aborted, you could have stranded chunks that won't be picked up. You can fix it the manual way that you described. I could code something to deal with it, but I probably won't be doing any updates for a while.

[tomaste]

Originally posted by quantum
If an encode is improperly aborted, you could have stranded chunks that won't be picked up. You can fix it the manual way that you described. I could code something to deal with it, but I probably won't be doing any updates for a while.

Oh? I was hoping to try RB Farm this week. Why the "no updates" ?

[quantum]

Because I don't feel like it. Why does it matter? Does the idea the program won't be changed every day turn you off from trying it?

[winny]

Superb program, thanks for all your efforts to date Quantum.

RBFarm works a treat from main PC, but I have a small problem with the second machine. It only creates small vaf and m2v files, whereas the main PC processes the files with no problems.

I am using eclcce 1.81 and cce 2.67.0.23.

Has anyone seen this before, or have any suggestions for a possible cause?

[wink]

Originally posted by b1ackmai1er
@Quantum

I must have had a hiccup with my RB Farm encode today because I had to restart it. It finished ok but when I Rebuilt with Rebuilder I got an error. I went back into the rbfarm directory and found that one segment had not finished encoding.

There was a farm_chunk_started0035 but no farm_chunk_finished0035 file. However there was a farm_finished file.

I restarted RBFarm and expected it to pick up this incomplete farm chunk and restart it, but it didn't.

I deleted the farm_finished file and restarted again but it still didn't pick it up.

Finally I succeeded by deleting both the farm_finished file and the farm_chunk_finished0035 file.

I thought about this for a while and then realized that this makes sense because RbFarm has no way to know if a farm PC has crashed or is still running that incomplete chunk. But it does know there are no more unstarted chunks so it writes a farm_finished file

So I wanted to suggest that you add a message that notifies the user in such a scenario. ie when you start or shut RB farm it checks if there is a farm_finished file. If there isn't it continues on respectively. If there is it should then check if it has any farm_chunk_startedXXXX files that are missing corresponding farm_chunk_finishedXXXX files and alert the user.

Hope this is clear and not to pedantic.

regards b1m1

I second that

[NobbyNobbs]

Originally posted by winny
Superb program, thanks for all your efforts to date Quantum.

RBFarm works a treat from main PC, but I have a small problem with the second machine. It only creates small vaf and m2v files, whereas the main PC processes the files with no problems.

I am using eclcce 1.81 and cce 2.67.0.23.

Has anyone seen this before, or have any suggestions for a possible cause?

When I tried it with four machines, the same happened on one of them.
I had an old version of AVISynth on it, uninstalled all versions, reinstalled 2.54 and everything worked like it should.

[tomaste]

Originally posted by quantum
Because I don't feel like it. Why does it matter? Does the idea the program won't be changed every day turn you off from trying it?

No no, I was just afraid you were "stopping" development. When you said "I could code something to deal with it, but I probably won't be doing any updates for a while." It sounded like something had put you off, and you were shelving it. I didnt want to fall in love, just to have my heart broken. ;)

Everyone needs some downtime! Have a good break.

On a side note, I will be trying this with a farm machine that has 4 xeon processors. Any experience with something like this?

[Joergen]

I'm still using 1.0 cause I dont need the features of the new versions! Quantum went much further than I needed :)

[280zx]

Would a slow powerline network be adequate for rbfarm? The network can only manage just under 3mbps. My main pc is a 3.2ghz p4 and the secondary pc is a 2.66ghz p4. I've already ordered a patch cable to connect them directly, through a router.

[Joergen]

I'd say homepna etc is not fast enough for your PC speeds. I havent tried yet but I'd suspect 10mbit starts to be too slow aswell.

[Joergen]

No such problems here. I've done about 10 encodes with RBFarm so far.

Check that you have the latest avisynth and ECLCCE on both PC's, ie the same exact setup on both. And check the paths and that none of the .m2v chunks contain the avisynth errors.

[wink]

i am pretty sure it's the same setup. The fact that it runs perfectly fine with progressive parts. I believe the movie play fine, just undersize. I can also check again later tonight.
by itself, both computer encode interlace material just fine. It's just a problem with network rbfarm & interlace material for me
btw, movie is queen of the damned
i found the prob. my careless mistake.

[quantum]

Are there any P4 Hyperthreading RB Farm users? If so, report the percentage of CPU used by RB Farm in comparison to the CPU DVD Rebuilder takes on the local PC.

[280zx]

I finally got around to setting this up, and it's very cool. With a 3.2ghz p4 and a 2.6ghz p4, the network traffic averages out at just over 13 mbps but it jumps around pretty wildly with 30mbps spikes.

Originally posted by quantum
Are there any P4 Hyperthreading RB Farm users? If so, report the percentage of CPU used by RB Farm in comparison to the CPU DVD Rebuilder takes on the local PC.

I'm a little late with this info since the cpu usage has been fixed in dvdrebuilder, but it's about the same now.

Anyway, nice work quantum, thanks.


Edit- RBfarm seems to have overloaded my router, a wired 10/100 linksys. After the backup my internet speed was about 1/4 what it's supposed to be. I had to restart my router to correct the problem. Bittorrents used to do the same thing until I switched to the azureus client.

Edit 2 - It only seems to have crashed my router the first time I used it. The last 3 backups went perfectly. Cool :D

[Joergen]

Just thought I'd add a note.

The new MPEG2Dec3dg.dll that comes with the http://www.doom9.org/Soft21/Decoders/decodefix110.zip package really does improve random access (HD usage in general) especially for RB Farm.

I remember reading about it earlier but didnt bother to update. But today I was listening to the HD of the serving PC clunking wildly still after only one network PC was still encoding and I thought "does it really need to trash around so much for a few megabits".

Apparently with the new mpeg2dec, it now doesnt! :)

Or am I just dreaming?

[djackson]

I had RBFarm 1.1 working perfectly and upgraded to 1.6 and am having probs on the "Farmed" PC's.

All encoding works fine on the server. When loading REBUILDER.ECL on the Farms, all chunks are read as 240 frames.

On server:
U:\ is source material (local drive)
V:\ is target and D2VAVS material (local drive)
d:\dvd-rebuilder\rebuilder.exe - RBFarms RB dir
v:\dvd-rebuilder\thegame\d2vavs\ - RBFarms D2VAVS dir
d:\cce sp trial\eclcce.exe - RBFarms ECLCCE dir

On Farms:
U:\ is source material (network drive)
V:\ is target and D2VAVS material (network drive)
v:\dvd-rebuilder\thegame\d2vavs\ - RBFarms D2VAVS dir
d:\cce sp trial\eclcce.exe - RBFarms ECLCCE dir

Any advice appreciated. Considered returning to v1.1, but overwrote all copies.

[quantum]

That has to be a coincidental problem. RB Farm is just feeding ECL's to CCE. Try encoding on the farm PC's with DVD Rebuilder. If that doesn't work, you have a problem somewhere else such as avisynth.

[djackson]

Thanks Quantum. Reinstalled AVISynth. All now works.

[Mutant_Fruit]

i'm just finishing setting up my 2 computer network.

An overclocked 2500+ and a standard speed 1800+. Lets test this baby out!

[Mutant_Fruit]

Yeah, there is a biggish performance hit when you use mapped drives.

Is there any way to share the drives/files across a network without having to map them using windows? Even mapping my local drive to M: and then (using the computer with the drive physically attached to it) i try and use DVD-RB to prepare, pointing it towards the M:, it goes slowly.

[Mutant_Fruit]

I had an idea. If RBFarm could be coded so that it can read files from an FTP server.

Basically, i just download a free FTP server app, then use the server to set the address of the D2VAVS folder to be something like ftp://192.168.0.1/D2VAVS and then i make both computers look to that for the folder.

That would stop the slow access with the drive remapping in windows. I hope i explained that well.

[Joergen]

FTP doesnt support random access in the same way true drive mapping does. Also for localhost drives, as mentioned in this tread, use the subst command instead. It maps your drive to a new letter with absolutely no slowdown (old optimized DOS command).

[Mutant_Fruit]

I didn;t know that about FTP, thats that idea down the chute.

And i didn't realise that the subst command meant no slowdowns. I thought it was because he was too lazy to map the drives normally (or wanted them mapped a different way for something else). I'll give it a shot now.

Thanks for the tip.

[Mutant_Fruit]

erm... explain exactly how i use subst to set up my D: to share between 2 computers. This computer i'm on, and one other.

[Joergen]

Oh, I thought since you mentioned prepare or rebuild being slow (which it is if you use the network mapping on localhost drives) you wanted a faster method, which subst provides.

You cant subst via a network. So the best way is to say subst the D: of PC#1 to O: and setup O: as the paths in dvd-rb, then map D: of PC#1 as O: on PC#2 (network mapping).

Now when you prepare and rebuild on PC#1 it will be as fast as if the O: was a real drive.

Preparing or rebuilding OVER the network will of course be slow since it's all about HD speed, and you'd need a gigabit network to be fast enough (and even then access times are several times as slow).

[Mutant_Fruit]

I understand now. subst is only used on my main computer. I thought subst was supposed to be used on all the computers, and i just couldn't get it to work on my second PC. Bah, i'm an idiot.

I got subst workin on this PC, and the preparing time is back down to 7mins, nice and zippy. Thanks for the pointers. I'm getting the hang of this now :) Great proggy btw.

Is there any way to donate via bank transfer?

[Joergen]

Paypal is sad in that most people in the nordic countries and most of europe dont use credit cards (like me) because you can always go around the corner to the ATM machine. And many shops dont even accept them due to a % going to the card company and cause of fraud :(

[Mutant_Fruit]

How do i go about setting up RB Farm and my computers to encode multiple movies in one night. Is "Batch mode" supported in RB Farm 1.6?

I tried running multiple instances, but EclCCE won't allow that. I don't want to set it up incorrectly, and end of overwriting each finished show as it starts the next.

[Mutant_Fruit]

bump. any ideas? From the looks of it, this feature isn't supported yet. But maybe i'm just stupid...

[Joergen]

Sorry, I dont think its supported. I've never needed a batch job myself as I like to play around with settings and tweak the output.

[Mutant_Fruit]

Ah well, looks like i have to wait for it to be supported.

In the mean time, could an option be added to turn off the farm computers when no more parts are available to encode. No point in my second computer being on all night when it finishes encoding about 1am...

[quantum]

Updated to version 1.7. This build adds batch processing functionality.

I wasn't going to bother adding this until I realized you can't do a batch job (unless I'm mistaken) in DVD RB unless you do one click mode. This means DVD RB will run a prepare phase wether you like it or not and any customizations you make with ECL Tweaker or ECL Optimizer, etc., are overwritten.

So now you can run the prepare phase on multiple jobs, make adjustments, then queue up the jobs in RB Farm. Even with one PC, which is all I use for encoding, this is an improvement.

This is not heavily tested so I wouldn't be surprised if some things have problems, especially the 'auto rebuild' feature, which won't work at all with multiple jobs.

[Joergen]

Sweet, Fruit got his wish! :cool:

[Mutant_Fruit]

LOL, i didn't expect that to happen.

I think its time i PM'ed the writer of DVD-RB and RB Farm and try and figure out some kind of bank transfer, as i really couldn't be arsed using paypal. I probably will be pretty busy for the next month or so (REALLY important exams coming up, that will decide the rest of my life!) so i won't be using my computer much over the next month or so (til the 25th of june, at 11:30a.m.).

Then i'll be back here, testing the latest version of RB and Farm, and organising some donations.

Now... if only i could recover that crashed harddrive...

[Joergen]

quantum: I know this might be impossible but please consider it.

I explained in the 30+ bug thread how a chunk got canceled via CCE first thus leaving a short .m2v.

Would it be possible to make RBFarm detect when CCE is canceled prematurely/first?

Or perhaps an option to check that all chunks are within a range and large enough, perhaps based on the average bitrate in the .ecl for each chunk?

[quantum]

Of course it's possible, but the problem won't happen if you cancel via RB Farm first. The only way to cause it is if you abort CCE first (just don't do it) or have a crash.

If you have a crash then you should know it. At that point get in there and clear it up manually. How many crashes do you have?

I don't know if I'll get to this since it's never been a problem for me, and it's not trivial to code.

[Joergen]

I dont crash (ever, knock knock knock) but somebody had canceled the job prematurely manually on one of the PC's while I was away.

If the PC would crash, it would reboot thus RBFarm would not write the finished file for that chunk.. no probs there.

[quantum]

I thought you were referring to the crash scenario. Regarding the manually aborted CCE scenario, now that I think about it, RB Farm would have a very hard time figuring out that an m2v was aborted versus a regular CCE finish. I would have to get into figuring out how many frames are in the m2v and compare to how many are reported in the AVS, both of which I have no idea how to do. In any case it would be hard work, possibly very hard work, and I'm sure I won't go down that road considering how rare the event should be.

Have you used the 'minimize CCE' checkbox in RB Farm? That should reduce the chances of accidentally closing the program. I use that myself.

Otherwise put a 'do not touch' sign on the PC. ;)

[winny]

Thanks for the added function Quantum, this is a very welcome surprise. I will start testing immediately!

[NobbyNobbs]

It looks like the link is not working properly at the moment. (The same goes for ECL Tweaker)

[acido]

@quantum
just a little problem i'm having with RB-FARM 1.6 (and 1.7) with latest builds of dvd-rb (0.51).
Yesterday i PREPARED with dvd-rb then started rb-farm on 2PCs with the following options :
- minimize cce
- auto encode
- auto rebuild ( only on the pc where i did the prepare )
.
First thing i noticed is that when rb farm started it did not miimize the DVD-RB window as usual, then at the end of the CCE steps, rb-farm was unable to auto rebuild and a popup window indicating it was unable to find dvd-rb window appeared (sorry i cannot remember the exact syntax).
Anyway after starting the rebuild manually on dvd-rb all went ok.
.
Are you getting the same?

[d62ks821]

It would be a fairly simple matter to remote administer the farm if rbfarm had the capability to accepted all options as command line params. For instance, if I launch rbfarm via telnet, pcexec, ssh... rbfarm with start up with the rebuilder.ecl path blank, and as such does not start processing.

[d62ks821]

Originally posted by chadp1a
Followed your advice on the sluggish network problems and it worked... thanks...

getting cce speed of 1.9+ on both pc's now :)


I have a similar problem. 2 nearly identical PC's the PROC on main is pegged at 100% while the PROC of the second only raises to 50-70%. Here is my config (main--switch--second):

ATH 1700+/512MB DDR ----- 100/10 switch ----- ATH 2100+/512MB SDRAM

Things I tried to debug this problem:

1) Stopped farm on main. Only slight increase in PROC utilization on second.

2) LAN utilization is only at 10% (6% main->second, 4% second->main), so i didn't think net connection/switch was a problem, but none the less ran some file copy experients. I could achieve 160% utilization by simultaneously copying a lG file main->second adn second->main. So clearly I'm at 100MB full duplex.

3) verfied that CCE is running at IDLE prority on both machines, and on the main machine CCE is consistantly running at around 97% of PROC utilization. ...so dont think problem is main side serving.

4) suspect problem was some drive mapping overhead, so mapped source rip and working drives locally on main. and GUESS WHAT!!! now PROC utilization on main is 50-70% as well! SO, I tried mapping the source rip drive and subst the working drive and PROC utilization was back to 100%. The inverse (subst the source rip and mapping the working) also was 100%.

CONCLUSION: Problem has nothing to do with my LAN. Problem appears to be some inefficiency (or extra wait states) with Windows XP when there is simulanteous activity on two different mapped drives.

[d62ks821]

Originally posted by d62ks821

CONCLUSION: Problem has nothing to do with my LAN. Problem appears to be some inefficiency (or extra wait states) with Windows XP when there is simulanteous activity on two different mapped drives.

One additional experiment. I move my source rip to the same physical HD as the working directory so I could run with only one mounted drive. AND running the main PC farm through 1 mapped drive and the second PC through 1 mapped drive, still resulted in poor PROC utilization (50-70%) on both PC's. I'm stumped.

[Joergen]

I've also noticed that since upgrading to a more recent .dll (decodefix110) and rbfarm 1.6 the fastest PC is only doing 60-70% of cpu time while with the old configuration of rbfarm 1.0 and previous mpeg2dec .dll it did 90-100%.

I'll try downgrading. I kept the 1.0 rbfarm somewhere.

[quantum]

Originally posted by acido
just a little problem i'm having with RB-FARM 1.6 (and 1.7) with First thing i noticed is that when rb farm started it did not miimize the DVD-RB window as usual, then at the end of the CCE steps, rb-farm was unable to auto rebuild and a popup window indicating it was If this was with 1.7 then I'm not surprised. Since I added the batch job capability, I haven't tested the 'auto rebuild' feature. RB Farm can't rebuild if you have more than one job, and maybe not even with one job (untested by me). I'll look at it when I have time.

The only way I can think of to get around the multiple-jobs and auto rebuilding issue is to accept only saved projects (rbd files) as input, instead of pointing to rebuilder.ecl.

If your problem is with 1.6, then it was likely a random failure. The way I coded RB Farm to remote control DVD Rebuilder is likely not to work 100% of the time.

[gnode]

Originally posted by acido
@quantum
just a little problem i'm having with RB-FARM 1.6 (and 1.7) with latest builds of dvd-rb (0.51).
Yesterday i PREPARED with dvd-rb then started rb-farm on 2PCs with the following options :
- minimize cce
- auto encode
- auto rebuild ( only on the pc where i did the prepare )
.
First thing i noticed is that when rb farm started it did not miimize the DVD-RB window as usual, then at the end of the CCE steps, rb-farm was unable to auto rebuild and a popup window indicating it was unable to find dvd-rb window appeared (sorry i cannot remember the exact syntax).
Anyway after starting the rebuild manually on dvd-rb all went ok.
.
Are you getting the same?

Same problem here on my farm!

[acido]

Originally posted by quantum
If your problem is with 1.6, then it was likely a random failure. The way I coded RB Farm to remote control DVD Rebuilder is likely not to work 100% of the time.

@quantum
Tried another time,
problem is solid with 1.6, i think something has changed in DVD-RB (for sure with 0.51b) and RB-FARM is no longer able to interface it.
As you can see gnode is having the same trouble, i'm not the only :) ...

[d62ks821]

Originally posted by Joergen
I've also noticed that since upgrading to a more recent .dll (decodefix110) and rbfarm 1.6 the fastest PC is only doing 60-70% of cpu time while with the old configuration of rbfarm 1.0 and previous mpeg2dec .dll it did 90-100%.

I'll try downgrading. I kept the 1.0 rbfarm somewhere.

How did your experiment go? I still have been unable to determine why I'm getting as little at 50% proc utilization on the networked farm PC. Other things I tried:

1) once CCE is running on the networked machine, I kill rbfarm on both PC's. Still have poor PROC untilization on networked PC. I don't think rbfarm is the problem.

2) ...next I kill the CCE on the main PC, so the only thing left running is CCE on the neworked PC. Still have poor PROC untilization on networked PC.

[gnode]

I have an 8 machine farm and strangely I can get 100% cpu utilization by running the xp performance monitor in the background on the slave machines. If I bring the performance monitor to the foreground the cpu utilization drops down to the 75% range. If I highlight the farm app, so that the performance monitor goes to the background, the cpu utilization goes back to 100% but not right away. Sometimes it waits for a new rb process to start before it goes back to 100%.

Give this a try and see if you can reproduce my results.


BTW my farm is a mix of amd and intel machines and the result is similar on both types.

[d62ks821]

Originally posted by gnode
I have an 8 machine farm and strangely I can get 100% cpu utilization by running the xp performance monitor in the background on the slave machines...

Give this a try and see if you can reproduce my results.


I presume you are referring to the performance tab on the task manager. Task manager runing, not running, forground, or background makes no difference.

I noticed that one of my disks was making alot of thrashing noise when I run via mapped network drive. So to try and eliminate the thrashing (and presumably the associated IO wait time that is likely causing the poor PROC utilization), I tried these things:

(1) I disabled system swap file, to eliminate it as a possible bottle neck. No difference. So re-enabled.

(2) Set swap file to fixed size of 2X my DRAM (i.e., 1GB). And then defragged boot disk. No difference.

(3) Defragged disk where my working directory resides. No difference.

[gnode]

d62ks821,
Are you saying that you repeated my experiment of having the task manager performance window running, not running etc. and saw no change in cpu utilization?

BTW, I'm using a 3com managed switch and I set cce on the master to idle as mentioned in page 3 of this thread.

gnode

[d62ks821]

Originally posted by gnode
Are you saying that you repeated my experiment of having the task manager performance window running, not running etc. and saw no change in cpu utilization?
correct. no change.

Originally posted by gnode
BTW, I'm using a 3com managed switch and I set cce on the master to idle as mentioned in page 3 of this thread.
master CCE is set to idle priority. And i'm using a netgear 10/100 switch. (however, see my ealier post, i doubt the network connection is the problem because I can reproduce poor PROC utilization on the master running by itself, with no slaves, by simply network mapping the working drive locally rather than using subst command)

[d62ks821]

Originally posted by d62ks821
I'm stumped

Ok, Major breakthrough here! By moving input and output files around, I was able to isolate problem to .mpv file output by CCE SP v2.50. About 30 seconds into the writing of the .mpv file, the disk starts to thrash unexpectedly. Looking around in the CCE menus I found: "/Options/Misc/Use Overlapped Output" which is checked. If I uncheck it, proc utilization gets much better and disk stops thrashing.

CCE on master outputing to network mapped drive on master:
With overlapped checked proc utilization is 50-70%
With it unchecked, pegged at 100% !!!

CCE on slave outputing to network mapped drive on master:
With overlapped checked proc utilization is 50-70%
With it unchecked, at 90% !!! (still don't undertand the 10% loss)

To clear this check box by default, set the following in the system registry:
"HKEY_CURRENT_USER\Software\Custom Technology\Cinema Craft Encoder SP\overlapped = 0"

[gnode]

The version of CCE I'm using (2.6x) doesn't have this setting. Disabled by default? I'll check if adding a key to the registry makes a difference.

Gnode

[gnode]

I changed to another 3com switch and now I get 100% cpu utilization on all 8 hogs on the farm!!


Still have the error when it tries to do auto rebuild!

Gnode

[lrosado]

When I run RBFARM on the second PC I get an error that states that it could not get the information. CCE Loads up but with the error and doesnt encode. I think it says something like "GdipLoadImageFromFile Failed :3". What could I do to get this fixed?

[lrosado]

anyone????

[gnode]

Did you do the map network drive correctly? Are the movie and working folders on the first machine set to sharing?

[lrosado]

Nevermind, go it to work, I thought I had installed the avisynth and I never did. I installed everything else but that, stupid me. Thanks to those who replied.

I got cce to do 3.0 on PC#1
I got CCE to do 2.0 on PC#2

thats 5x combined speed, awesome little program.

It did get stuck at the auto rebuild phase(only selected on the main PC), said it could not find the rebuilder window or something like that.

[Mutant_Fruit]

best not to use auto-rebuild. They way its coded, it doesn't always work due to the way it had to be coded. The best thing to do is save each project in DVD-RB, and after its compresed, rebuild it manually.

[Uppercuts]

I set up rbfarm on all three of my PC's. It works fine on the main PC but when I try to encode on client machines it gives me the "Can't Get File Attributes" error. Is there something i am doing wrong? Help will be greatly appreciated. Thanks...

[lrosado]

make sure all working paths (path to d2vavs) are the same on all rbfarms and that both the source and working directory is on the mapped drive. and that avisynth v2.54 is installed on all three machines along with CCE along with ECLCCE, if needed, on all three machines.

Infiniterb wrote

I did the prepare process on one PC, and have that drive mapped to Z:

RB's paths are the folllowing:
The source is Z:\JurassicPark
The Destination is Z:\VDWRKDIR

For DVD RB Farm:

On PC ONE

D2VAVS Path: Z:\VDWRKDIR\D2VAVS
ECLCCE path: C:\Program Files\ECLCCE\eclcce.exe

On PC TWO

D2VAVS Path: Z:\VDWRKDIR\D2VAVS
ECLCCE path: C:\Program Files\ECLCCE\eclcce.exe

You would just do the same for PC#3

[d62ks821]

Originally posted by gnode
I changed to another 3com switch and now I get 100% cpu utilization on all 8 hogs on the farm!! Gnode
Have tried several things, and still can't peg CPU utilization at 100% on slave pc's.

Netgear router switch - 85-90%,
3Com switch - same.
no switch (crossover) - worse (65-75%), likely due to the fact that for some reason I can only get windows to run half duplex this way, despite setting both computer at fixed 100 FD in the driver advanced settings.

Using Intel 8255x based ethernet adapters in both machines.

[gnode]

Originally posted by d62ks821
Have tried several things, and still can't peg CPU utilization at 100% on slave pc's.

Netgear router switch - 85-90%,
3Com switch - same.
no switch (crossover) - worse (65-75%), likely due to the fact that for some reason I can only get windows to run half duplex this way, despite setting both computer at fixed 100 FD in the driver advanced settings.

Using Intel 8255x based ethernet adapters in both machines.

Have you tried changing the various parameters on the ethernet cards, such as flow control, etc.? I set all parameters on all machines the same.

When I had some intel ethernet cards in some machines and 3com cards in others I had problems. Things improved when I went to all 3com cards and a 3com switch.

Also, one of my slaves is an Nforce2 machine with an addon pci ethernet card and for some unknown reason it has trouble staying at 100% cpu utilization. My master is also an Nforce2 machine. All the other machines both AMD and Intel all pretty much peg at 100%.

For kicks I tried my 8 port etherfast switch -- poor cpu utilization all around.

[wink]

For my problem, I came to the conclusion that XP is the culprit. At first I use 2 computers, one 2k, one xp with sp2 (beta) the cpu utilization on both computers are high. As I add my laptop (XP no service pack) to the equation, the cpu on laptop is around 60-70%. I tried change the file serving from 2k to XP, both low cpu utilization. But when I update xp to sp2 (beta), the cpu usage jump up to around 95% during encoding process. Just want to let you guys know. As always, BACK UP your system first just in case MS screw up ;)

[old-hack]

I've just successfully used RBFarm 1.7 and am really impressed.

I thought I'd throw out a couple of things that I found were not totally obvious and clear about configuring the source computer and Farm computers - especially for a new user to RBFarm.

Please, no flames about reinterating what a lot of you may think was obvious.

1. On the source computer where the files are "prepared", you must prepare the DVD files using a mapped drive to the DVD source
"VIDEO_TS" folder. Do not prepare your files using the drive letter of your hard drive on the source PC. This mapped drive letter must be the same in the "source path" box in DVDRebuild on both the source PC and the Farm PCs.

2. You must create a mapped drive to the "working folder" created on the source PC. This mapped drive letter will be the drive letter you put in the "working path" box in DVDRebuilder on both the source PC and the Farm PCs.

3. The DVD Rebuilder directory in RBFarm is the local PCs path to DVDRebuild for the Farm PCs. You do not create a mapped drive for this path.

4. The "Rebuilder ECLs" path in RBFarm is the "D2VAVS" folder beneath the drive letter mapped to the "working folder" on the source PC. When you browse the mapped "working folder" drive, you will see the "D2VAVS" folder. Once you select this folder and press "OK", the correct path will appear in the "Rebuilder ECLs" path in RBFarm. The mapped drive letter\"D2VAVS" path must be the same on all Farm PCs.

As a suggestion and for clarity, I think the field labeled "Rebuilder ECLs path" in RBFarm shoud say "Source D2VAVS path" or "Mapped Source D2VAVS path".

Thanks.

[DVDCake]

I was pointed to this thread by cdfreaks.com.

Before I start setting up all this software to use RB Farm can you guys tell me if this is a mpeg2 encoding util or some kind of shrinking app for DVDs? I need to encode from AVI or DV to mpeg2 so I can author to DVD. I would like to farm out the encoding to my rack of pc's.

Thanks,

DVDCake

[wmansir]

Sorry, this program can't help you. It works with DVD ReBuilder, which only does DVD->DVD-R.

[DVDCake]

Ah, but I still see a use for it.

Thanks!

[The Belgain]

I've just tried this one. Seems great. Is jdobbs planning on integrating this into DVD-RB? It would be nice to have everything in a single tool, and not have this go down the DVD2SVCD route (I need two plugins on top of the original program, and all the programs it frontends to in order to get DVD2SVCD to do what I want it to do).

Great work though. On a slightly unrelated topic, does anyone know of any all in one switch/wireless access point/router/ADSL modem things with Gigabit ethernet and 'g' wireless? All the ones I've seen are only 100 MBit LAN. This seems to be reaching the limits of the 100 MBit connection on PC1 (with 3 other PCs connected to it on the farm).

[gnode]

Anybody know how to get around the XP PRO's 10 machine limit for concurrent access to a shared network drive when using RBFarm?

[quantum]

I don't know, but please post some numbers if you're doing a 10 station encode :)

[gnode]

To get around XP's 10 machine limit,

I installed standard server 2003 on the master machine.

I've tested the setup and everything works well and I've confirmed that the 10 machine limit is no more!

Now to get the full cluster up and running.

When complete, I'll post some results.

[gnode]

Using 14 machines (fastest is 2.3 GHz, slowest is 900 MHz) I re-encoded (two passes)a 7.5 GB movie in 75 minutes. This is about the same time it took with 10 machines.

There are two issues. The first is that at the end there is always one of the slower machines encoding a large chunk that extends the time by 10 to 15 minutes, i.e., all the other machines are done.

Quantum -- a suggestion, Could you modify the code so that the chunks are reordered from largest to smallest so that the biggest ones are done first and then when all encoding is complete, they are automatically restored to their original order for rebuild?

The second issue is that as the number of machines is increased, the efficiency (i.e., cpu utilization) decreases. I terminated the encode on the master machine and the cpu utilization on all the slaves returned to near %100. Clearly the master cannot encode and handle all the slaves at the same time. I am going to try using a slower machine as the master so that I can get the benefit of my fastest machine.

[quantum]

Originally posted by gnode
Quantum -- a suggestion, Could you modify the code so that the chunks are reordered from largest to smallest so that the biggest ones are done first and then when all encoding is complete, they are automatically restored to their original order for rebuild?
Possibly. I'll look into it.

[tange1]

hey, I'm having a odd problem with RB farm. Maybe someone can tell me why this is happening. I just discovered rb farm so maybe im making some stupid noob mistake but a read through of the thread didn't help.

My problem is this - I'm encoding a movie that I stripped some extra's out of. The files at start are roughly 5 gigs. RB farm runs fine on a host machine, and one other machine. (the host being the faster machine) Everythign runs great, and the files are rebuilt. But in the end I have 2.7 gigs of files instead of the 4ish I'd normally expect out of rebuilder.

What am I doing wrong?

[wink]

Originally posted by tange1
hey, I'm having a odd problem with RB farm. Maybe someone can tell me why this is happening. I just discovered rb farm so maybe im making some stupid noob mistake but a read through of the thread didn't help.

My problem is this - I'm encoding a movie that I stripped some extra's out of. The files at start are roughly 5 gigs. RB farm runs fine on a host machine, and one other machine. (the host being the faster machine) Everythign runs great, and the files are rebuilt. But in the end I have 2.7 gigs of files instead of the 4ish I'd normally expect out of rebuilder.

What am I doing wrong?

I would say mapped drive. The same drive letter has to be on both machine. Try open .avs file with window media on second machine and see what happen

[tange1]

the drive letter is the same F: in this case. Must the drive letter also match for the location of rebuider. I mean, in my 2nd machine (non primary) i have the rebuidler location set to local location on the machine.

[geffroman]

I have just completed 9 discs in batch and have come up with a suggestion I think would be very simple to implement and be extremely helpful to automate the process...

If we could SAVE rebuilder.ecl paths so that the same paths were there the next time we open the program we could automate workflow dramaticly. Imagine:

You map to a drive were you keep your movie data... In this drive you have folders for:

\movie_1\d2vavs
\movie_2\d2vavs
\movie_3\d2vavs

You place these locations in RBFARMs paths... SAVE the paths and close the program...

Run your PREPAREs in DVD-RB with working location being the movie_1; movie_2 and movie_3 locations... when finshed you simply start RBFARM and walk away... No running around to 2, 3 or 5 computers to place paths in RBFARM everytime you run a movie...

Take it a step further... You have RBFARM in your start up group on an old computer you never use anyway... after you and run your PREPARES you just turn on the computer and walk away... It boots up... starts RBFARM with the saved generic locations and does the work...

Or you could have RBFARM loaded on several computers and just Netmeeting from your main computer in for a moment to start RBFARM with the presaved paths from one locations...

Or the best one yet... You set up RBFARM on a schedule to run at night automatically... It launches at Midnight with the presaved paths... runs any movies in the folders... Next morning the work is done... A you had to do was PREPARE movies before you go to bed...

Bottomline is that I found the biggest time waster involved was the setting up of paths on several computers... A more advanced approach would be for the main computer to be able to feed all available clients the path information automatically... but this would take a little more programming than the above method...

If anyone knows who I could use the registry to save paths that would work too...

Thanks... Jeff

[wgw]

I am getting "Run-time error 53 file not found" using RBFarm 1.6 and 1.7 with CCE SP Retail. When I select the CCE folder it says "Warning - eclcce.exe not found in folder". Then when I try to encode I get the run-time error. Eclcce should not be required with the retail version. Encoding works fine with DVD-RB by itself, but I would like to use RBFarm if possible.

[trance1977]

WGW -

After you select the CCE folder, and you get the error message about eclcce, just type "cctsp.exe" at the end of the path. Then try running rbfarm, it should work fine.

[wgw]

Thanks Trance! Had to delete the RB Farm settings from the registry to get that change to stick after closing RB Farm, but it's working great now.

[trance1977]

No problem, glad to help. :D

[huesage]

Hello to all,

I have read the whole thread and this is becoming very confusing for me. Does anyone have a dummies guide to correcty setting the working folders and mapped drives?

I am having alot of trouble visualizing where which files have to be. I was under the impression that on my Server machine I had to create a working folder and use subst to give it a drive name, but when i try to share that drive so that the client machine can map and use that drive... it says that it does not exist and is not possible to do.

I am sorry if I am the dumbest in the group right now... but I am at a loss and would love to run RB-Farm so any help would be greatly apreciated.

If you can help and find it easier to use IM please add me to your list:

MSN: hue_sage@hotmail.com
YAHOO: huesage@yahoo.com
AOL: transfersk8

I thank you all in advance for all your help.

[huesage]

Hey guys,

Thank you all so much for just juumping on and helping me out, very nice of all of you. This happens to be one of the features which I asked for from jdobbs early on when i donated money.

Oh well, hopefully one day it will be better documented and make more sense to me.

[gnode]

At the end of an encode there is always one of the slower machines encoding a large chunk that extends the time in some cases as much as 45 minutes i.e., all the other machines are done.
Quantum -- a suggestion, Could you modify the code so that the chunks are reordered from largest to smallest so that the biggest ones are done first and then when all encoding is complete, they are automatically restored to their original order for rebuild?

[geffroman]

I ran about 30 movies using RB Farm 1.7... It was wonderful...

But now my remote computers all run at a faster than normal speed and I end up with a seriously undersized movie. If I run RBFarm from the main computer only it works fine.

The only thing that I did that may have caused this was I upgraded all the systems to DVD-RB V63. I had been using DVD-RB V56. Everything worked fine but V63 caused sound problems so I went back to DVD-RB V56.

Now I have the problem. I have uninstalled everything including Avisynth, CCE SP, ECLCCE, DVD-RB, RBFarm and then REINSTALLED everything again but the problem won't go away.

Computers that used to run at a speed of about 1.8 now run at 3.0 and finished movie is way undersized... Any thoughts...?

Could this be movie specific? I just tried two other movies that seem to work correctly. Discs that demonstrated the problem are:

Roswell Season 2
Taxi Season 1
Adventures of Winnie the Pooh
Rio Grande

[quantum]

Maybe something changed in the ECL format of the new version? I'm going to get around to updating RB Farm in the next couple of weeks so I'll try it with the new version.

[Thor263]

This might be a dumb/noob question, but will RBFarm also work with Quenc? Since it is slower than CCE, this would be ideal.

[huesage]

I finally managed to set this up correctly and ran backup for K-PAX NTSC.

Server machine: Athlon64 3000+/1GB PC3200 OCZ/Asus K8V-Deluxe
Client machine: Dell Inspiron 600m/Pentium - M 1.4Ghz/384 MB RAM

Both machines are running:

- avisynth 2.55
- DVD-RB 0.64a
- CCE SP 2.67.00.23
- ECLCCE 1.81
- MPEG2Dec3dg.dll (tried the new DGDecode.dll and failed)
- RB-Farm 1.7 (what a wonderful app)

average encode speeds:

- Server: 2.20
- Client: 1.60

total encoding time was of 111 minutes. This movie is 2 hours long and has extras that were encoded at halfd1 with 50% bitrate stolen.

This whole process was flawless exept for the AUTO-Rebuild, it did not function correctly. I had opened DVDRB on server, ran prepare, then opened RB-Farm and ran it with RB open in background. Another instance of RB was opened so I am about to run another job with RB closed while Farm is running.

I hope this info means something to someone and helps in some way shape of form.

[Thor263]

Originally posted by Thor263
This might be a dumb/noob question, but will RBFarm also work with Quenc? Since it is slower than CCE, this would be ideal.

Maybe I should rephrase this - I don't know how difficult it would be to implement this type of change to RB Farm, but it would be nice to be able to use it with Quenc as well. I'm assuming CCE is easier since you can generally just feed it parts of the .ecl file whereas you would have to do more w/ Quenc. I might have that bit wrong though.

Would be a nice feature!

[jwu42]

Originally posted by geffroman
I ran about 30 movies using RB Farm 1.7... It was wonderful...

But now my remote computers all run at a faster than normal speed and I end up with a seriously undersized movie. If I run RBFarm from the main computer only it works fine.

Could this be movie specific? I just tried two other movies that seem to work correctly. Discs that demonstrated the problem are:

Roswell Season 2
Taxi Season 1
Adventures of Winnie the Pooh
Rio Grande

Seeing the same thing here... (The MAtrix R1) DVD RB .67 with CCE 2.5 (other stuff latest - AVI SYnth, DGDecode, DVD Decrypter, etc.)

Client machine is running at over 4 (Pentium M 1.8) and the encodes are winding up close to 3GB. Both are mapped to Y (Source) and Z (Working). Are there any known issues for copying the VIDEO_TS folder to a drive instead of using the virtual drive as Y: ?

[geffroman]

Nobody replies... nobody cares... Farm is dead!

[NobbyNobbs]

If you look a few posts up, quantum says he is going to update the program in a few weeks time.:)

[MaceWindu]

First of all: This tool is AWESOME! THX Quantum!

I'm using this on seven machines, everything works just fine except for one thing: After (roughly) 70 minutes the six clients stop encoding as if there's nothing to do anymore. The server still encodes several chunks which takes up to 3 hours. Somehow this ruins the benefit of the farm-concept... :-)

Any ideas how to fix this issue?

If there's anything what i could to in order to help, i would gladly do that!

Regards,
MaceWindu


_______________________
The specs of my "farm":

- 6 clients (P4, 2,8 GHz, 512 MB Ram, 40 GB HDD, Gigabit-NIC)
- 1 "server" (P4, 2,8 GHz, 1024 MB Ram, 40 GB + 160 GB HDD (one disk containing the sources and one with the working folder, Gigabit-NIC)

OS is WinXP SP2

[geffroman]

Originally posted by NobbyNobbs
If you look a few posts up, quantum says he is going to update the program in a few weeks time.:) That was more than 2 months ago... and the many unanswered posts I have go much farther than that...

Originally posted by MaceWindu
First of all: This tool is AWESOME! THX Quantum!

I'm using this on seven machines, everything works just fine except for one thing: After (roughly) 70 minutes the six clients stop encoding as if there's nothing to do anymore. The server still encodes several chunks which takes up to 3 hours. Somehow this ruins the benefit of the farm-concept... :-)
I am cracking up... You say everying works fine EXCEPT ONE THING... It don't work... That's a pretty big thing...

Clearly Farm is a nice idea that I too would like to use... But lack of responses and software development tell the story...

[pluonk]

This is my first time using RBFarm (using v1.7) and I'm having the exact same problem with geffroman: On Mullholland Drive, I got a 2.7GB movie, for the most part messed up (the duration was correct on PowerDVD but few parts of the movie were actually playable). I also noticed the much greater CCE speed on the client PC (#2), even though that PC is slower than the server. I read quantum's post about a possible change in the .ecl that Rebuilder saves and I noticed the following differences (with DOS's FC command):

***** D:\CD RIPS\D2VAVS\REBUILDER.ECL
vaf_out=1
timecode=0x0000000
opv_q_factor=20
***** D:\DVD RIPS\!DVDRB\WORKING\D2VAVS\REBUILDER.ECL
vaf_out=1
opv_q_factor=20
*****

***** D:\CD RIPS\D2VAVS\REBUILDER.ECL
opv_brate_min=0
opv_brate_max=7200
vbr_bias=25
***** D:\DVD RIPS\!DVDRB\WORKING\D2VAVS\REBUILDER.ECL
opv_brate_min=0
opv_brate_max=7046
vbr_bias=25
*****

***** D:\CD RIPS\D2VAVS\REBUILDER.ECL
vbr_brate_min=0
vbr_brate_max=9000
seq_endcode=0
***** D:\DVD RIPS\!DVDRB\WORKING\D2VAVS\REBUILDER.ECL
vbr_brate_min=0
vbr_brate_max=8808
seq_endcode=0
*****

***** D:\CD RIPS\D2VAVS\REBUILDER.ECL
quality_prec=16
video_type=4
vid_file0=O:\DVD RIPS\!DVDRB\WORKING\D2VAVS\V02000400005001.m2v
***** D:\DVD RIPS\!DVDRB\WORKING\D2VAVS\REBUILDER.ECL
quality_prec=16
timecode=0x0000000
q_char_f=16
tc_offset=0
v_filter=0
v_filter_val=6
pict_name=
pict_type=3
pict_level=255
video_type=16
vid_file0=O:\DVD RIPS\!DVDRB\WORKING\D2VAVS\V02000400005001.m2v
*****

[...]

***** D:\CD RIPS\D2VAVS\REBUILDER.ECL
opv_brate_min=0
opv_brate_max=7200
vbr_bias=25
***** D:\DVD RIPS\!DVDRB\WORKING\D2VAVS\REBUILDER.ECL
opv_brate_min=0
opv_brate_max=7046
vbr_bias=25
*****

***** D:\CD RIPS\D2VAVS\REBUILDER.ECL
vbr_brate_min=0
vbr_brate_max=9000
seq_endcode=0
***** D:\DVD RIPS\!DVDRB\WORKING\D2VAVS\REBUILDER.ECL
vbr_brate_min=0
vbr_brate_max=8808
seq_endcode=0
*****

***** D:\CD RIPS\D2VAVS\REBUILDER.ECL
quality_prec=16
video_type=4
vid_file0=O:\DVD RIPS\!DVDRB\WORKING\D2VAVS\V02000600007001.m2v
***** D:\DVD RIPS\!DVDRB\WORKING\D2VAVS\REBUILDER.ECL
quality_prec=16
timecode=0x0000000
q_char_f=16
tc_offset=0
v_filter=0
v_filter_val=6
pict_name=
pict_type=3
pict_level=255
video_type=16
vid_file0=O:\DVD RIPS\!DVDRB\WORKING\D2VAVS\V02000600007001.m2v
*****

***** D:\CD RIPS\D2VAVS\REBUILDER.ECL
vaf_out=1
timecode=0x0000000
opv_q_factor=20
***** D:\DVD RIPS\!DVDRB\WORKING\D2VAVS\REBUILDER.ECL
vaf_out=1
opv_q_factor=20
*****

[....]

***** D:\CD RIPS\D2VAVS\REBUILDER.ECL
quality_prec=16
video_type=4
vid_file0=O:\DVD RIPS\!DVDRB\WORKING\D2VAVS\V02000900010001.m2v
***** D:\DVD RIPS\!DVDRB\WORKING\D2VAVS\REBUILDER.ECL
quality_prec=16
timecode=0x0000000
q_char_f=16
tc_offset=0
v_filter=0
v_filter_val=6
pict_name=
pict_type=3
pict_level=255
video_type=16
vid_file0=O:\DVD RIPS\!DVDRB\WORKING\D2VAVS\V02000900010001.m2v
*****

***** D:\CD RIPS\D2VAVS\REBUILDER.ECL
vaf_out=1
timecode=0x0000000
opv_q_factor=20
***** D:\DVD RIPS\!DVDRB\WORKING\D2VAVS\REBUILDER.ECL
vaf_out=1
opv_q_factor=20
*****

***** D:\CD RIPS\D2VAVS\REBUILDER.ECL
opv_brate_min=0
opv_brate_max=7200
vbr_bias=25
***** D:\DVD RIPS\!DVDRB\WORKING\D2VAVS\REBUILDER.ECL
opv_brate_min=0
opv_brate_max=7046
vbr_bias=25
*****
The .ecl file at D:\CD RIPS\ was made with Rebuilder 0.56
The .ecl file at D:\DVD RIPS\!DVDRB\WORKING\ was made with Rebuilder 0.67

(edited out)

I didn't actually try RBFarm 1.7 with Rebuilder 0.56 yet (PC's doing other things right now) but will do as soon as possible. In the meantime, I'd appreciate any info/advice on the problem, at least to avoid wasting hours in test encodes.

[b1ackmai1er]

Check that the avisynth path on all computers is the same.

i.e. One of the automatically generated avisyth script for my last encoded movie looks like this:

file: V01000000001001

#------------------
# AVS File Created by DVD Rebuilder
# VOBID:01, CELLID:01
#------------------
LoadPlugin("D:\Program Files\AviSynth 2.5\plugins\DGDecode.dll")
mpeg2source("X:\DVD-RB\WORKPATH\D2VAVS\V01.D2V")
trim(0,2051)
ConvertToYUY2(interlaced=true)
AudioDub(BlankClip())

The paths for loadplugin and mpegsource must be the same on all PC's.

regards b1m1

[MaceWindu]

Just a small update on my farm...

I did another encoding using 3 passes (for testing...). 4 machines were used in this test. (3 x P4 2,8, 512 MB; 1 x P4 2,8, 1024 MB)

Software specs:

- WinXP SP2
- AviSynth 2.54
- RB Farm 1.7
- DVD ReBuilder 0.57
- CCE SP 2.66 + EclCCE

Prepration time: 4 minutes
Total encoding time: 113 minutes (fastest system was the host with 98 minutes)
Rebuilding time: 12 minutes
All together: roughly 2 hrs and 10 minutes

So basically this thing IS WORKING FINE!

I will do some more testing, maybe i can find a clue, why my host took that much longer last time...

Cheers,
MaceWindu

[pluonk]

I found the cause of the undersized movie (and the abnormally high encoding speed in the client PCs) and it turns out it's a stupid error on my part (rbfarm's readme.txt and the posts in this thread don't talk about this explicitly, which I believe is why this problem is not that uncommon).

We, users, are told to be careful with paths: all programs/dlls as well as dvdrb's working folder must be the same on all PCs. I made sure they were (using DOS's subst to map drives correctly etc) EXCEPT for one thing: the .vob files that the .d2v files point at. For all encodes, I rip the dvd to an iso image, then load that image to a virtual drive. So the paths in the .d2v files generated by dvdrb point at that virtual drive, which of course only exists in the PC where you run the 'Prepare' phase.

The solution to this problem is, again, drive mapping. What I did is:
[list=1] Created a virtual dvdrom drive on PC #1 and assigned Y: to it.
Mapped that drive on PC #2 as Y:
Created a virtual dvdrom drive on PC #2 and assigned Z: to it.
Mapped that drive on PC #1 as Z:[/list=1]
This way, you can load the iso image to a virtual drive on either PC and the other PC would automatically refer to it by the same letter. It is a good idea to always use one of these two virtual drives (even if you have more) when doing farm encoding, to avoid future problems. Of course, if you rip your dvds to a VIDEO_TS folder (instead of an iso image) you can just put that folder on the same drive with dvdrb's working folder (I just don't do that because I don't always have the necessary disk space on the working drive).

I hope this helps everybody else that has the same problem.

[kappa]

I have used RB-farm with success as a lot of other people have. It's philosophy is very simple, yet effective. Problems start to arise when the user makes mistakes conscerning the paths and when programs change versions (eg now that Rebuilder can use CCE SP 2.70, I cannot use RB-farm).

While the first point is up to every user to consider/resolve, the second one has one feasible (and sustainable) solution: implementation of farm logic into Rebuilder itself. If one has already resolved all the problems with using various encoders, settings etc like it has been done for DVD-Rebuilder, it is IMO trivial to implement a farm function where other machines are able to join in at will at the encoding process.

I imagine it would take Jdobbs no more that a couple of hours to do this and enhance the usability of rebuilder.

[geffroman]

Originally posted by kappa
I imagine it would take Jdobbs no more that a couple of hours to do this and enhance the usability of rebuilder.

It appears to me that jdobbs is very resistant to ANY third party involvement beyond supporting of encoders. Over time he is frustrated with the ever changing CCE/ECLCCE issue. Has even stated had he known what he was in for that he might not have undertaken the project. Additionally, he has NO interest in solving issues related to preprocessing of DVD backups even though most people do preprocessing.

I too would love Farm support in RB. I just don't think it will happen.

[kappa]

Originally posted by geffroman
It appears to me that jdobbs is very resistant to ANY third party involement beyond supporting of encoders.
Just to make sure that my point got across correctly, I wasn't referring to support of a third party program. I am talking about original source code inside DVD-Rebuilder which is summed up by the algorithm

1. Encode remote avs stream.
2. Generate necessary files.
3. Inform a progress file/directory of what has been done.
4. Restart ftom step 1.

This would be the algorithm even the server machine would use. It would just use a progress file/directory (like "farm_status") so as not to re-encode a segment undertaken by another host. At the end, the server would automatically rebuild the final dvd.

Most of the work behind this (steps 1 and 3) is already available to him. After all, if I remember correctly, the real magic of DVD-Rebuilder, which has taken up most of Jdobbs energy, is the authoring part.

I hope that our donations alleviate some of his frustration and keep his spirits up.

[geffroman]

Your post might go better answered in the RB threads... Not sure jdobbs will see it here... I am sure everyone in this thread agrees that Farm implementation is what we want... after all this is a Farm thread...

Much luck on getting jdobbs interest in this! :)

[acido]

Hello quantum,
unfortunately without a new version of eclcce that will support CCE-SP 2.70 trial i am unable to use RB-FARM with this CCE-SP version .

It seems that CCE-SP 2.70 trial can handle ecls directly without the need of eclcce. The only benefit would be to run the encoder minimized.

Can you have a look and if possible support cce-sp 2.70 in your application directly?

Taken from: ECLCCE Thread (http://forum.doom9.org/showthread.php?s=&threadid=46664&perpage=20&pagenumber=15)

Anyway, remember, for CCE 2.70, the only benefit of using EclCCE will be the minimize-to-tray and credits bitrate tweaking features....

[DvPs]

@acido

i think you can use rf-farm with cce 2.7. you just have to insert a comment in the rebuilder ecl file (serach fpr rb and cce 2.7)
and in farm you have to set the whole path.
for example:

c:\programme\cce sp 270105\cctsp.exe

mfg DvPs

[acido]

@DvPs

Thanks a lot for your suggestion i will try it soon, but i'm afraid that way the CCE-SP install dir mus be the same on all the computers... I have one running from C drive and another from E.

Thx again.

[juk]

I tried to backup one of the Seinfeld Season 4 DVDs with DVD Rebuilder and RB Farm using two computers and the final movie size amounted only to 3GB. I think this is the exact same problem that geffroman described in one of his posts. Has anyone that has had this problem been able to resolve it?

The version of the software I'm using is the following:

DVD Rebuilder v. 0.7 (I also tried using v. 0.64a)
CCE 2.50 SP
AviSynth 2.55
ECLCCE 1.81
RB Farm 1.7
DGMPG 1.0.12 (DGDecode.dll)
Decomb 5.2.1


Thanks.

[acido]

@DvPs

I have tried your suggestion, but CCE SP 2.70 gave me an error saying the ECL was not compatible.
Will stay with CCE 2.67 waiting for changes in RBFarm

[DvPs]

@acido

hi

do you have changed something in the rebuilder.ecl that is in your working directory?

when not, do this:

open your cce. fiel-open-video file.
save an ecl.
open the ecl with the editor (i don't know the exact name in englisch, the text editor).

the first line should look something like that:

; CCE SP Trial Version -- Encoder Control List

now open your rebuilder.ecl file

the first two lines should look look like this:
; Cinema Craft Encoder SP -- Encoder Control List
; Created by DVD Rebuilder

now replace "Cinema Craft Encoder SP" with "CCE SP Trial Version".
then it should work....

i hope i could help you

mfg DvP

[acido]

Ok i was able to use RBFARM with cce-sp 2.70.01.05 directly without the need of eclcce.

After setting up DVD-RB correctly to generate the right ecl, the only thing that needs to be done is to paste the full path to the cce exe in RBFARM 'ECLCCE path (or CCE basic)' . (e.g: c:\program files\dvdrb\cctspt2701.exe )

Regards.

[xriderbc]

So, is development on this completely dead? That's too bad- it works well. It would be great if the rebuild automatically setting worked so that overnight setups could complete.

[acido]

Yes, agree
but unfortunately quantum seems disappeared ... :confused:

[tijgert]

Problem 1: My secondary PC does about 1/10th of a second on 1 chunk... I'm guesing that's wrong (DUH)

Problem 1: Fixed, re-prepared with RB78 from Z: drive to make it all standard.

My setup:

PC1 has the DVD ripped in d:\dvd\sg1
Z: is subst for D:
RB78 prepare's from Z:\DVD\sg1 with working dir Z:\DVD

PC2 has Z: mapped from PC1\\d:\

On both PC's RBFarm points to the correct (same) files (I'm trying to encode using 2.70.02 with pointing directly to the .exe).
.ecl paths on both PC's is Z:\DVD\D2VAVS

Running RBfarm in Autoencode first on PC1 puts it to work after about 10 seconds, on PC2 same.

Problem 2: I have to click encode on the CCE window on both PC's, is that normal? (doubtfull and most annoying)

[b1ackmai1er]

@tijgert

Did you give write access to the drives/directories shared?
Sharing by default is read-only I think.

Does chunk 1 and 3 complete sucessfully?

You should not have to press encode to start CCE working. Something is flaky there.

regards b1m1

[tijgert]

My network is most unsecure, I can do anything to anything anywhere.
If anybody gains access AND guesses my pw, I'm fubar :)

Chunk 1 and 2 seem to be just fine (won't be sure untill the rebuilding).

I re-prepared everything with RB78 from Z: drive and now it's encoding just fine.
My laptop is the Main PC doing chunk 1 and my Game PC (while not gaming) is helping out on chunk 2 now.

On both PC's CCE wanted me to press encode before it started.
Could this be because I am using CCE SP Trial 2.70.02.00 with RBFarm when it's supposed to be using an older version?

Am I supposed to tick the 'minimize CCE' tick-box on RBFarm?
(I ticked it now, I'll see in an hour what it does, but seems very unlikely that minimizing will auto-click the encode button).

Update: Game PC just finished chunk 2 and asked again to encode chunk 3, unworkable for overnight encoding.

Did anybody ever come across this situation with this setup? (see previous post). Anybody have any clues as to what it might be or how to work around it? (short of switching back to eclcce and/or older CCE SP 2.67)

Any help is greatly appreciated!

Fixed: on RBFarm select 'minimize CCE' and all goes well... well, until rebuilding so far :-)