Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > (HD) DVD, Blu-ray & (S)VCD > DVD & BD Rebuilder

Reply
 
Thread Tools Search this Thread Display Modes
Old 21st April 2004, 00:27   #1  |  Link
quantum
Registered User
 
Join Date: Nov 2002
Location: USA
Posts: 528
RB Farm v1.7 - server farm for faster DVD Rebuilder encodes

HERE is a link to the current version of RB Farm. The zip file size should be 25,585.

This thread describes the development and testing of RB Farm, an assistant program to DVD Rebuilder. RB Farm allows multiple PCs to work together on a network to perform a faster CCE encode than could be performed with a single PC.

1.7 - Added batch processing functionality. You can queue up multiple jobs and they will be processed in order.

1.6 - Upgraded auto encode functionality. You no longer need to press the encode button on a single PC to start the encode process. This should allow one click functionality on a farm. You should be able to press "prepare" in DVD Rebuilder and RB Farm will automatically do the rest.

1.5 - Fixed some problems with "rebuild automatically"

1.4 - Added auto rebuild
The "auto rebuild" check box will cause the current instance of RB Farm to attempt to rebuild the project after encoding using DVD Rebuilder. You should only select this option on the PC where DVD Rebuilder is installed and where you ran the prepare phase. DVD Rebuilder should be running with the settings for the current project correctly chosen, i.e., audio streams deselected, etc. The mode should not be one click mode, and "suppress warning prompts" should be selected.

1.3 - Added auto encode functionality
The "auto encode" check box will cause the current instance of RB Farm to periodically detect if a job is ready to be encoded in the d2vavs directory. If a job is ready, RB Farm will automatically join in and process the next available chunk.

1.2 - Added support for CCE Basic. You have to type in the EXE file name manually on the "path to ECLCCE" line.

1.1 - Fixes CCE 2.5 support


Last edited by quantum; 29th May 2004 at 23:11.
quantum is offline   Reply With Quote
Old 21st April 2004, 00:51   #2  |  Link
Joergen
Guest
 
Posts: n/a
Rocking

Mirror:
rbfarm.zip 1.7 25KB

Last edited by Joergen; 30th May 2004 at 17:23.
  Reply With Quote
Old 21st April 2004, 19:52   #3  |  Link
EddieTH
Unindicted Co-conspirator
 
Join Date: Jan 2004
Location: Lost Somewhere In Time
Posts: 62
I'm trying out a modified version of this now. I have 3 PCs to encode with, but the CPU and RAM vary widely between the 3 (P4 2.4GHz/768MB DDR, Celeron 1.7GHz/128MB DDR, & Athlon 1GHz/256MB SDRAM). Since taking 1/3 for each PC would actually take longer because the fastest machine encodes at more than 5x the speed of the slowest, I did a test encode with one of the segments and compared the speeds on each of the PCs. Then I calculated what percentage of frames they should each encode to get done at roughly the same time. This would be the fastest I could possibly encode with this configuration. Finally, I determined how many segments to encode with each PC based on the frame count.
EddieTH is offline   Reply With Quote
Old 21st April 2004, 21:08   #4  |  Link
quantum
Registered User
 
Join Date: Nov 2002
Location: USA
Posts: 528
I thought more on this, but I didn't post anything simply because I lost interest. To me, I'm not that speed hungry. I don't mind queueing up batch jobs and waiting overnight. But with said, I'll offer a bit more:

The latest version of CCE does not have the memory leak problem that has plagued previous versions when loading multiple AVS files (see the CCE forum). I downloaded the trial version and indeed it looks fixed. Thus you should be able to simply load the ECL file into each instance of CCE (on separate PCs), then trim out half (or whatever) jobs in CCE (with ECLCCe) and encode. This would eliminate all the nonsense steps listed above.

Additionally, I figured I could code up a quick program in a few hours or so to take jdobbs rebuilder.ecl file and feed each chunk to eclcce on each machine. This would have the benefit of automatic load balancing and reduce the manual steps down to nothing. Maybe I'll get to this if someone else doesn't do it, but as jdobbs said, it's probably better to wait until the engine is 100% stable before worring about things like this.
quantum is offline   Reply With Quote
Old 21st April 2004, 21:19   #5  |  Link
EddieTH
Unindicted Co-conspirator
 
Join Date: Jan 2004
Location: Lost Somewhere In Time
Posts: 62
I agree, it's not a big priority, especially since my second machine only encodes at about 1/3 the speed of the first, and the third machine is significantly slower than that. I also noticed that the speed for each of the remote encodes dropped by about 10% when they were all running while the local encode slowed down by around 20%. Even when the third machine finished (it got less than planned because of the size of individual segments), the other 2 encodes didn't get that much better speed.

I'll probably play around with this more since I have a lot of time to do it, but I probably won't do anymore 3 part encodes since the gain from the last machine is minimal. My main interest is in seeing how much performance DVD-RB can manage with multiple PCs, which I would say is a fairly significant amount with the right setup, but not enough to be worth the trouble in many cases.
EddieTH is offline   Reply With Quote
Old 21st April 2004, 21:29   #6  |  Link
Joergen
Guest
 
Posts: n/a
You could use the slowest computer to serve the source and the faster ones to compress. With DMA 100mbit LAN the lan transfer itself shouldnt eat up more than 1-5% of CPU time.

But if you have old 10Mbit LAN cards the CPU usage might be 20-50% depending on the chip and driver and speed of transfer.. but mostly those cards are history.
  Reply With Quote
Old 21st April 2004, 22:03   #7  |  Link
EddieTH
Unindicted Co-conspirator
 
Join Date: Jan 2004
Location: Lost Somewhere In Time
Posts: 62
I probably could, but there's another issue with the slow machine. It's a laptop. I'm not real excited about the idea of heating it up for that long. I have another machine I'm building with a case and mobo that was given to me. It will probably end up being about the same speed as the first one (or maybe a little faster), so I may try out 3 PC encoding again when it's finished.
EddieTH is offline   Reply With Quote
Old 21st April 2004, 22:06   #8  |  Link
Joergen
Guest
 
Posts: n/a
I have 3 desktop PC's in this room and the makings of two more stored away, I only need 2 ATX cases with PSU. If somebody writes a handy extension to dvd-rb or jdobbs gets around to supporting this idea I'll surely slap together those puters
  Reply With Quote
Old 21st April 2004, 22:08   #9  |  Link
quantum
Registered User
 
Join Date: Nov 2002
Location: USA
Posts: 528
Maybe I will fire up something if you two want to test it. Shouldn't take too long..
quantum is offline   Reply With Quote
Old 21st April 2004, 22:24   #10  |  Link
quantum
Registered User
 
Join Date: Nov 2002
Location: USA
Posts: 528
If you two want, send me a PM with an email address where I can send beta copies. I can't post attachments here. I should have something ready tonight.. (famous last words of every programmer)
quantum is offline   Reply With Quote
Old 21st April 2004, 22:27   #11  |  Link
EddieTH
Unindicted Co-conspirator
 
Join Date: Jan 2004
Location: Lost Somewhere In Time
Posts: 62
If you make it I'll certainly put it to work. I work from home so I'm in front of my computers for a great deal of the day. It's nice being able to combine hobbies and work.
EddieTH is offline   Reply With Quote
Old 21st April 2004, 22:27   #12  |  Link
Joergen
Guest
 
Posts: n/a
Sounds great, but could you host it somewhere instead so we can dl.
  Reply With Quote
Old 22nd April 2004, 01:06   #13  |  Link
quantum
Registered User
 
Join Date: Nov 2002
Location: USA
Posts: 528
I completed a very rudimentary prototype. It works but it's too basic to post and I expect problems so I'd like to get an email correspondence going with testers to avoid cluttering up jdobbs forum.

@EddieTH I sent you a copy via email.
@Joergen I sent you a PM with the location and details.

Anyone else interested in testing this PM me (preferrably with an email address) and I'll send you a copy.
quantum is offline   Reply With Quote
Old 22nd April 2004, 21:47   #14  |  Link
Joergen
Guest
 
Posts: n/a
Any news here? I'll be trying the first version now with a disc and two puters.
  Reply With Quote
Old 22nd April 2004, 21:48   #15  |  Link
quantum
Registered User
 
Join Date: Nov 2002
Location: USA
Posts: 528
I guess it's been tested enough.

HERE is a link to the current version of RB Farm. I'm not sure how long I'll keep it hosted there, but considering how small it is, I guess I'll keep there for a little while. If jdobbs or doom9 consider it worthy, maybe it will be hosted in a more permanent spot.

Big thanks to EddieTH for testing and debugging. Good testers are hard to find. Of course big thanks to jdobbs for all his great work.

Instructions are in the readme. Comments are welcome.
quantum is offline   Reply With Quote
Old 22nd April 2004, 21:53   #16  |  Link
Joergen
Guest
 
Posts: n/a
Rocking! Sorry for not being able to test it earlier.

Hmm thats interesting, the error 9 seems to come from removing streams in shrink. So I'll set the targetsectors higher and overshoot, then use shrink to remove the huge-hiney subs.

Last edited by Joergen; 22nd April 2004 at 22:32.
  Reply With Quote
Old 22nd April 2004, 23:16   #17  |  Link
Joergen
Guest
 
Posts: n/a
The WORKING and SOURCE path, it would seem, really limits the use of RBFarm for most (since how many people have the source computer with a path that is so far down the alphabet that the other PC dont have it reserved.)

It might be obvious to some but not all:

TIP:

1) On the source PC (the one that has the dvd files, and want to run DVDRB on and do the PREPARE), go to the network neighborhood and map the drive (ie D: ) to H: and use H:\ as the WORKING PATH in DVD-RB.

2) Then on PC#2 map the same drive (D: @ PC#1) as H: on PC#2.

There you have the same WORKING path on both PC's easily.

You need to do the same to the source path, so placing the source in ie D:\MOVIE\VIDEO_TS on PC#1 and then using H:\MOVIE\VIDEO_TS as the source. And of course H:\D2VAVS and H:\MOVIE\VIDEO_TS can be found on all PCs once you mount the drive.



edit: names of paths

Last edited by Joergen; 22nd April 2004 at 23:40.
  Reply With Quote
Old 23rd April 2004, 01:21   #18  |  Link
quantum
Registered User
 
Join Date: Nov 2002
Location: USA
Posts: 528
Good tip. Of course anyone wanting to set up an encoding farm should be expected to do some amount of network setup.

Is it otherwise working and performing as expected?
quantum is offline   Reply With Quote
Old 23rd April 2004, 01:42   #19  |  Link
Joergen
Guest
 
Posts: n/a
Yes I've now tested it back and forth and a PC#2 encoding off PC#1 takes no CPU power from PC#1, and PC#2 is able to do about -0.05 (in CCE speed) of what it does off its own HD, and PC#1 can do exactly as it normally does.

It's working really great so far! I can get about 3.20 combined speed compared to the half with either computer doing it alone, really helps since alone each PC can take about 5 hours with a movie.

I can see it now, just map H: on a few puters, and hit the RBFarm exe on the desktop! Hey presto, 6x CCE speed

Thanks alot quantum. I'll buy two ATX cases soon and build barebone systems just for RBFarming!

Old McDonald had a farm.. hiiiae hiiiae hey...

Last edited by Joergen; 23rd April 2004 at 02:02.
  Reply With Quote
Old 23rd April 2004, 03:55   #20  |  Link
quantum
Registered User
 
Join Date: Nov 2002
Location: USA
Posts: 528
Good news. I'm curious what happens if you put a 3rd and a 4th PC on there. The data transfer rate of CCE is not that high, around 1mb/sec for 1x. Your 10/100 LAN can do 10 times that and your hard drive 20 times that. So maybe 4 or even more would be okay.

The theoretical limit for your 10/100 LAN would be around CCE 10x combined speed according to my rough guess.

That's pure speculation so let us know what happens :-)
quantum is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 18:43.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.