RB Farm v1.7 - server farm for faster DVD Rebuilder encodes

[quantum]

HERE (http://www.connforum.com/rbfarm.zip) is a link to the current version of RB Farm. The zip file size should be 25,585.

This thread describes the development and testing of RB Farm, an assistant program to DVD Rebuilder. RB Farm allows multiple PCs to work together on a network to perform a faster CCE encode than could be performed with a single PC.

1.7 - Added batch processing functionality. You can queue up multiple jobs and they will be processed in order.

1.6 - Upgraded auto encode functionality. You no longer need to press the encode button on a single PC to start the encode process. This should allow one click functionality on a farm. You should be able to press "prepare" in DVD Rebuilder and RB Farm will automatically do the rest.

1.5 - Fixed some problems with "rebuild automatically"

1.4 - Added auto rebuild
The "auto rebuild" check box will cause the current instance of RB Farm to attempt to rebuild the project after encoding using DVD Rebuilder. You should only select this option on the PC where DVD Rebuilder is installed and where you ran the prepare phase. DVD Rebuilder should be running with the settings for the current project correctly chosen, i.e., audio streams deselected, etc. The mode should not be one click mode, and "suppress warning prompts" should be selected.

1.3 - Added auto encode functionality
The "auto encode" check box will cause the current instance of RB Farm to periodically detect if a job is ready to be encoded in the d2vavs directory. If a job is ready, RB Farm will automatically join in and process the next available chunk.

1.2 - Added support for CCE Basic. You have to type in the EXE file name manually on the "path to ECLCCE" line.

1.1 - Fixes CCE 2.5 support

[Joergen]

Rocking :)

Mirror:
rbfarm.zip (http://koti.mbnet.fi/haupt/rbfarm.zip) 1.7 25KB

[EddieTH]

I'm trying out a modified version of this now. I have 3 PCs to encode with, but the CPU and RAM vary widely between the 3 (P4 2.4GHz/768MB DDR, Celeron 1.7GHz/128MB DDR, & Athlon 1GHz/256MB SDRAM). Since taking 1/3 for each PC would actually take longer because the fastest machine encodes at more than 5x the speed of the slowest, I did a test encode with one of the segments and compared the speeds on each of the PCs. Then I calculated what percentage of frames they should each encode to get done at roughly the same time. This would be the fastest I could possibly encode with this configuration. Finally, I determined how many segments to encode with each PC based on the frame count.

[quantum]

I thought more on this, but I didn't post anything simply because I lost interest. To me, I'm not that speed hungry. I don't mind queueing up batch jobs and waiting overnight. But with said, I'll offer a bit more:

The latest version of CCE does not have the memory leak problem that has plagued previous versions when loading multiple AVS files (see the CCE forum). I downloaded the trial version and indeed it looks fixed. Thus you should be able to simply load the ECL file into each instance of CCE (on separate PCs), then trim out half (or whatever) jobs in CCE (with ECLCCe) and encode. This would eliminate all the nonsense steps listed above.

Additionally, I figured I could code up a quick program in a few hours or so to take jdobbs rebuilder.ecl file and feed each chunk to eclcce on each machine. This would have the benefit of automatic load balancing and reduce the manual steps down to nothing. Maybe I'll get to this if someone else doesn't do it, but as jdobbs said, it's probably better to wait until the engine is 100% stable before worring about things like this.

[EddieTH]

I agree, it's not a big priority, especially since my second machine only encodes at about 1/3 the speed of the first, and the third machine is significantly slower than that. I also noticed that the speed for each of the remote encodes dropped by about 10% when they were all running while the local encode slowed down by around 20%. Even when the third machine finished (it got less than planned because of the size of individual segments), the other 2 encodes didn't get that much better speed.

I'll probably play around with this more since I have a lot of time to do it, but I probably won't do anymore 3 part encodes since the gain from the last machine is minimal. My main interest is in seeing how much performance DVD-RB can manage with multiple PCs, which I would say is a fairly significant amount with the right setup, but not enough to be worth the trouble in many cases.

[Joergen]

You could use the slowest computer to serve the source and the faster ones to compress. With DMA 100mbit LAN the lan transfer itself shouldnt eat up more than 1-5% of CPU time.

But if you have old 10Mbit LAN cards the CPU usage might be 20-50% depending on the chip and driver and speed of transfer.. but mostly those cards are history.

[EddieTH]

I probably could, but there's another issue with the slow machine. It's a laptop. I'm not real excited about the idea of heating it up for that long. I have another machine I'm building with a case and mobo that was given to me. It will probably end up being about the same speed as the first one (or maybe a little faster), so I may try out 3 PC encoding again when it's finished.

[Joergen]

I have 3 desktop PC's in this room and the makings of two more stored away, I only need 2 ATX cases with PSU. If somebody writes a handy extension to dvd-rb or jdobbs gets around to supporting this idea I'll surely slap together those puters :)

[quantum]

Maybe I will fire up something if you two want to test it. Shouldn't take too long..

[quantum]

If you two want, send me a PM with an email address where I can send beta copies. I can't post attachments here. I should have something ready tonight.. (famous last words of every programmer)

[EddieTH]

If you make it I'll certainly put it to work. I work from home so I'm in front of my computers for a great deal of the day. It's nice being able to combine hobbies and work. :cool:

[Joergen]

Sounds great, but could you host it somewhere instead so we can dl. :o

[quantum]

I completed a very rudimentary prototype. It works but it's too basic to post and I expect problems so I'd like to get an email correspondence going with testers to avoid cluttering up jdobbs forum.

@EddieTH I sent you a copy via email.
@Joergen I sent you a PM with the location and details.

Anyone else interested in testing this PM me (preferrably with an email address) and I'll send you a copy.

[Joergen]

Any news here? I'll be trying the first version now with a disc and two puters.

[quantum]

I guess it's been tested enough.

HERE (ftp://ftp.connforum.com/rbfarm.zip) is a link to the current version of RB Farm. I'm not sure how long I'll keep it hosted there, but considering how small it is, I guess I'll keep there for a little while. If jdobbs or doom9 consider it worthy, maybe it will be hosted in a more permanent spot.

Big thanks to EddieTH for testing and debugging. Good testers are hard to find. Of course big thanks to jdobbs for all his great work.

Instructions are in the readme. Comments are welcome.

[Joergen]

Rocking! Sorry for not being able to test it earlier.

Hmm thats interesting, the error 9 seems to come from removing streams in shrink. So I'll set the targetsectors higher and overshoot, then use shrink to remove the huge-hiney subs.

[Joergen]

The WORKING and SOURCE path, it would seem, really limits the use of RBFarm for most (since how many people have the source computer with a path that is so far down the alphabet that the other PC dont have it reserved.)

It might be obvious to some but not all:

TIP:

1) On the source PC (the one that has the dvd files, and want to run DVDRB on and do the PREPARE), go to the network neighborhood and map the drive (ie D: ) to H: and use H:\ as the WORKING PATH in DVD-RB.

2) Then on PC#2 map the same drive (D: @ PC#1) as H: on PC#2.

There you have the same WORKING path on both PC's easily.

You need to do the same to the source path, so placing the source in ie D:\MOVIE\VIDEO_TS on PC#1 and then using H:\MOVIE\VIDEO_TS as the source. And of course H:\D2VAVS and H:\MOVIE\VIDEO_TS can be found on all PCs once you mount the drive.

:helpful:

edit: names of paths

[quantum]

Good tip. Of course anyone wanting to set up an encoding farm should be expected to do some amount of network setup.

Is it otherwise working and performing as expected?

[Joergen]

Yes I've now tested it back and forth and a PC#2 encoding off PC#1 takes no CPU power from PC#1, and PC#2 is able to do about -0.05 (in CCE speed) of what it does off its own HD, and PC#1 can do exactly as it normally does.

It's working really great so far! I can get about 3.20 combined speed compared to the half with either computer doing it alone, really helps since alone each PC can take about 5 hours with a movie.

I can see it now, just map H: on a few puters, and hit the RBFarm exe on the desktop! Hey presto, 6x CCE speed :)

Thanks alot quantum. I'll buy two ATX cases soon and build barebone systems just for RBFarming!

Old McDonald had a farm.. hiiiae hiiiae hey... :D

[quantum]

Good news. I'm curious what happens if you put a 3rd and a 4th PC on there. The data transfer rate of CCE is not that high, around 1mb/sec for 1x. Your 10/100 LAN can do 10 times that and your hard drive 20 times that. So maybe 4 or even more would be okay.

The theoretical limit for your 10/100 LAN would be around CCE 10x combined speed according to my rough guess.

That's pure speculation so let us know what happens :-)

[DDogg]

You guys define the word, "geek" :D

[Joergen]

Originally posted by DDogg
You guys define the word, "geek" :D

Why, thank you :o ;)

quantum: I'll try with 3 puters tomorrow. The third one being able to do about 2.5x on its own. So I can reach about 6x I think. Then I'll bring a Pentium-M 1.8GHz laptop home from work that should help me reach around 7.5x :D

If the HD starts to be too slow or other bottlenecks appear, I can see it by looking at the cpu usage on each system.

[int 21h]

@quantum: Can you explain more what your app is doing? Is it just copying the directory as you did in the beginning? Would you be willing to release the source for it? I have some pretty decent ideas on how to improve it (with complete automation on the nodes), and I have a pretty good hand in VB...

[quantum]

Originally posted by int 21h
@quantum: Can you explain more what your app is doing? Is it just copying the directory as you did in the beginning? Would you be willing to release the source for it? I have some pretty decent ideas on how to improve it (with complete automation on the nodes), and I have a pretty good hand in VB...
It analyzes the output from the prepare phase of DVD Rebuilder and figures out how many chunks will be encoded. Then each instance of the program picks the next available chunk, parses the rebuilder.ecl to pick out the appropriate ecl, and encodes it. The instance creates a flag in the d2vavs directory which indicates the chunk has been reserved for processing so the other instances don't collide with each other.

I probably won't release the code anytime soon and I plan on making a few more tweaks.

@all: For those having CCE encoding speed issues, you could try RB Farm and see if that makes a difference. You can run it on only one PC. If my code happened to make a difference I'll gladly turn it over to jdobbs, although I'm pretty sure our code is 99% the same.

[chadp1a]

I tried RB-Farm and it works perfect. My problem is on pc#1 it get cce speed of about 1.45 and on pc#2 cce speed on average is .60

Both Pc#1 and Pc#2 are identical computers in terms of processor, memory, hardware..........

When using only dvd-rb on one pc I get speeds of 2.0+

The router I'm using is very old and I notice there are tons of collisions going on. Could the collisions be a problem maybe?

[quantum]

A few things to look at:

I'm not sure how fast your LAN must be to take advantage of 2 PC's. A 10mb LAN might be fast enough, but I'm not sure. I have 100mb which translates to 10 megabytes per second. You should test your LAN speed by copying large files between PCs and measuring the speed.

Check you have UDMA enabled on all IDE controllers. You should be using less than 10% of your CPU when copying large files. Having this set in PIO mode might work when doing a single local encode but would likely cripple a multiple network encode.

Maybe a 10/100 router or hub would help.

Otherwise I don't know what to suggest. You'll have to experiment. Let us know if you figure it out.

[Joergen]

If the network has collisions it means its a 10Mbit half-duplex network that has to wait for packets to pass back and forth. Since CCE reads and writes at the same time, half-duplex will mean your get delays from only moving in either direction at one time.

I have a 100mbit full duplex switch (switch being the key word) that doesnt limit the network in such ways.

[chadp1a]

So a new router would solve the problem then right?

I have a new linsys 5port 10/100 swich/router that I haven't tried on it yet.

Thanks for the info guys!

[Joergen]

edit: nm it was an old version of eclcce on one PC that made MPV files instead of M2V :o

[Joergen]

Originally posted by chadp1a
So a new router would solve the problem then right?

I have a new linsys 5port 10/100 swich/router that I haven't tried on it yet.

Thanks for the info guys!

Dont forget to use CAT5 cables between the router and PCs, and set the network adapters to 100mbit FULL DUPLEX in windows.

[Joergen]

Observations of the effectiveness of RBFarm:

-3pass CCE job with same project.

1PCs = 334 minutes

2PCs = 232 minutes

3PCs = 125 minutes

Instead of 5 hours (as I did often earlier) I can do it in 2! :)

I noticed that shutting down (disabling) Norton Antivirus 2004 helps about 15%. That software is weird, apparently it monitors network traffic vigorously. Shutting down Zonealarm didnt give significant gain.

[chadp1a]

@all
2 passes with CCE using RB Farm here.....

1 pc - 134 minutes

2 pc's - 87 minutes

@Joergen

Followed your advice on the sluggish network problems and it worked... thanks...

getting cce speed of 1.9+ on both pc's now :)

@Quantum
Nice addon for DVD-RB. Thanks to both you and Jdobbs

[P3gasus]

I have a question about rbfarm. I didn't understand if it assigns chunks statically or dynamiccally. I mean: if I have a p3 500 and an amd 2200xp on a lan will the chunks be assigned to the computers proportionally to the encoding speed of both? Or will they be assigned half to pc1 and half to pc2? I think it should not be difficult to implement dinamical cell assignement with a client/server application using for example java rmi or simply sockets. I could try to think about something if you think it can help.

Bye

PS. This prototype I have in mind does NOT use RB for encoding but it calls eclcce for encoding directly.
Tell me if this could cause problems.
PPS. I couldn't test rbfarm by myself because currently the link is not working.

[Joergen]

Originally posted by P3gasus
I have a question about rbfarm. I didn't understand if it assigns chunks statically or dynamiccally.
PPS. I couldn't test rbfarm by myself because currently the link is not working.

The "assignment" is the simplest and most effective method. Each PC picks up the next available chunk and encodes it, then skips the ones that are done or reserved by other PCs and encodes another chunk.

And RBFarm is standalone and only needs the prepared files in d2vavs and eclcce.

This way (and I have inbalanced setups) the fastest PC did 55% of the chunks, the middle speed (that was hosthing also) about 30% and the lowly slow puter only 15% :)

I'll try to host it someplace for all.

[P3gasus]

Originally posted by Joergen
The "assignment" is the simplest and most effective method. Each PC picks up the next available chunk and encodes it, then skips the ones that are done or reserved by other PCs and encodes another chunk.

Cool :D. That's simple but indeed effective. I can't wait to try it.
Thanks.
Bye
Pegasus

[Joergen]

Here you go all. RBFarm 1.1

rbfarm.zip (http://rbfarm.is.dreaming.org/rbfarm.zip) 24KB Dont use "save as", just left-click.

Thanks again quantum for making DVD-RB almost as fast as a transcoder for me :D

edit: Updated to 1.1

[quantum]

Originally posted by Joergen
Observations of the effectiveness of RBFarm:
-3pass CCE job with same project.
1PCs = 334 minutes
2PCs = 232 minutes
3PCs = 125 minutes Decent looking numbers and confirmed by chad1a. I wonder how much difference a switch makes over a hub in this case. I know the switch is supposed to be more efficient in theory but a real world test of a 10/100 switch vs. a 10/100 hub would be interesting.

[Joergen]

I only have switches so I cant test that one. Tomorrow I can test how a slower 700mhz PC does at the end of a 10mbit line, and if using two physical drives for read (source) and write (d2vavs dir) makes a difference.

[HanSolo00]

Pretty cool... eventually you could top out at 1 pc per chapter/chunk and get complete VTS encodes down to the 10-15 minute mark.

If I was still the IT manager at my old workplace you can guarantee I'd be trying that after hours on the LAN... just imagine how many projects you could queue up and complete overnight:)

[quantum]

I think there is a maximum you can achieve with this technique that would be hit before you could have 1 PC per chunk. You're manipulating files directly off the server so you're limited by the serving machine's ability to send and receive.

My original guess was you could achieve a combined CCE 10x on a 100megabit LAN. If each PC is running at 2x, this would be 5 PCs.

However now I'm thinking this may be higher if you have full duplex switches instead of hubs.

[EddieTH]

Originally posted by DDogg
You guys define the word, "geek" :D
A geek friend of mine had a similar reaction. ;) He also thought it was pretty cool. :p

[b1ackmai1er]

No luck getting this going yet.

Keep getting the following error in CCE:

i:\dvd-rip\d2vavs\farm_chunk0001.ecl is not a ECL for Cinema Craft Encoder SP

Is this possibly caused by the .ecl extension being lower case?


These are my rbfarm paths:
--------------------------

i:\dvd-rip\d2vavs\
d:\program files\dvd-rip\eclcce\eclcce.exe

The contents of my eclcce.ini file:
-----------------------------------
[EclCCE]
CCE=D:\Program Files\Custom Technology\Cinema Craft Encoder SP\cctsp.exe

The contents of farm_chunk0001.ecl
----------------------------------

[item]
title=V01000000001001
aud_out=0
vaf_file=I:\DVD-RIP\D2VAVS\V01000000001001.vaf
aud_file=I:\DVD-RIP\D2VAVS\V01000000001001.mpa
file_focused=0
packet_size=2048
width=720
height=576
frame_rate_idx=3
cbr_brate=6000
vbr_brate_avg=6253
vbr_brate_min=0
vbr_brate_max=9000
seq_endcode=0
dvd=1
half_width=0
half_height=0
lum_level=0
aspect_ratio=3
gop_m=3
gop_nm=4
gop_hdr=12
seq_hdr=1
all_closed_gop=0
fix_gop_length=0
samples_per_sec=44100
stereo=2
brate_idx=7
crc=1
progressive=0
alternate_scan=1
intra_dc_prec=2
encode_mode=0
vmode=2
fast_mode=0
quality_prec=24
vid_file0=I:\DVD-RIP\D2VAVS\V01000000001001.m2v
vid_file1=I:\DVD-RIP\D2VAVS\V01000000001001.m2v
vid_out=1
vaf_out=1
video_type=4
timecode=0x0000000
opv_q_factor=20
opv_brate_min=0
opv_brate_max=9000
vbr_bias=25
vbr_pass=4
use_filter=0
filter_val=6
non_linear=1
top_first=0
mpeg1=0
mpeg1_cps=1

[file]
name=I:\DVD-RIP\D2VAVS\V01000000001001.avs
frame_first=0
frame_last=12562
encode_first=0
encode_last=12562

Hope someone can help.
Am very excited about this program.

regards b1m1

[Joergen]

Originally posted by quantum
My original guess was you could achieve a combined CCE 10x on a 100megabit LAN. If each PC is running at 2x, this would be 5 PCs.


You forget one thing. You can put two 100mbit lan cards into the server pc, pull out one RJ45 to two separate 100mbit switches, and divide the node PCs to two groups for each switch. This way you not only get dedicated 100mbit for each group, but the (low cost) switches are better off not having to serve as much traffic :D

But theoretically you can calculate that the max bandwidth of a DVD is 1mbit, and lets say the returning encoded video is 0.4mbit (4000kbit). So on a PC that does 2X in CCE you suck out 2mbit and return 0.8mbit, and add some room for overhead bandwidth you can say its 3mbit per PC. A 100mbit network can, in reality, do about 50mbit per direction on low cost adapters. That leaves room for 16 PC's of 2X caliber.

Today I'm testing how much encoding on the serving PC hurts the network performance. As CCE seems to run like a High Priority program (everything else stands still for seconds) eventhough it is in normal priority.

[quantum]

@b1ackmai1er
I believe this happens when you run the prepare phase of DVD RB with something other than options->cce options->cce sp (2.66+)

For instance if you have cce basic selected you'll get this error. You should also get this error if you try to drag and drop rebuilder.ecl into CCE.

Also make sure you have the latest eclcce.

[quantum]

Originally posted by Joergen
But theoretically you can calculate that the max bandwidth of a DVD is 1mbit, and lets say the returning encoded video is 0.4mbit (4000kbit). So on a PC that does 2X in CCE you suck out 2mbit and I don't know if theory works out in practice. I based my figures on my network bandwidth meter. It shows me pulling around 10megabit from the server when encoding at 1x. This is about 10% capacity of my 10/100 LAN. The return stream is much less, around .5megabit.

[Joergen]

Results:

I used an XP2100+ PC with WinXP and 1GB of DDR333 RAM, and two physical 7200rpm drives to serve the source and d2vavs. Then an AMD64 3200+ to encode off that PC. When the server is not encoding the AMD64 encodes at 100% CPU usage, but when the server is also encoding the AMD64 usage graph fluctuates from the low 40% to around 90% constantly.

This leads me to think it would be best to serve on the slowest (400-900mhz) PC on the network with a decent HD, or if available, dedicate a faster PC just for serving. Dedicated serving can get the final edge out of RBFarm.

quantum: is there a low priority switch for CCE or ECLCCE? That might help the serving PC keep up with the underlying tasks.

[b1ackmai1er]

Hi Quantum,

Thanks for such a quick reply.

Yes I am using CCE 2.50 and have set Rebuilder to use it using ECLCCE 1.8b. This combination works fine for rebuilder.

I am confused. Are you saying that I cannot use this version with RBfarm?

My understanding was that the whole purpose of the ECLCCE was to make versions such as 2.50 work with ECL files, otherwise there would be no need to specify the ECLCCE path and we would be running CCE directly with each farm.ecl file.

Are you able to clarify that for me?

Thanks again.

Regards b1m1.

[quantum]

Originally posted by Joergen
quantum: is there a low priority switch for CCE or ECLCCE? That might help the serving PC keep up with the underlying tasks. Yes, run eclcce (which will launch CCE),

help->about->eclcce tab
CCE Process Priority->idle

You can also change it manually in the eclcce.ini file.

[Joergen]

Originally posted by quantum
Yes, run eclcce (which will launch CCE),

help->about->eclcce tab
CCE Process Priority->idle

You can also change it manually in the eclcce.ini file.

Excellent! I never noticed that tab (RTFM!) ;)

Well that seems to do it. The server is still doing 100% and the AMD64 is doing 100% as before. Now the 3-8% of CPU usage the server had from the others using its resources gets done FIRST and not last.

This might be a semi-important note for optimizing your RBFarm people
:helpful:

[b1ackmai1er]

Yes Quantum you are right!

I had RBfarm set to Rejig for preparation pass.

Will try again.

Damn I promised my wife I would be in bed before midnight tonight :(

Thanks b1m1