RB Farm v1.7 - server farm for faster DVD Rebuilder encodes

[quantum]

I don't think it will work with 2.50 yet, but I think I know what the problem is.

I'll post an update soon.

[quantum]

I updated the link on page 1 to version 1.1. This should allow CCE SP 2.50 to work with RB Farm. The way I was parsing the rebuilder.ecl file worked with CCE 2.66+ but not with 2.50. It should work with both versions now.

I still don't think it will work with CCE Basic, but I don't know for sure.

[Infiniterb]

For some reason this seems like it's moving insanely fast for my encodes.

I've got a 3 pass CCE job going, but the process to encode doesn't seem like it's doing anything. It just does the first pass extremely fast, then the second, then third.

I did the prepare process on one PC, and have that drive mapped to Z:

RB's paths are the folllowing:
The source is Z:\JurassicPark
The Destination is Z:DVDWRKDIR

For DVD RB Farm:

On PC ONE

D2VAVS Path: Z:DVDWRKDIR\D2VAVS
ECLCCE path: C:\Program Files\ECLCCE\eclcce.exe

On PC TWO

D2VAVS Path: Z:DVDWRKDIR\D2VAVS
ECLCCE path: C:\Program Files\ECLCCE\eclcce.exe

Is there anything else I'm missing?

[Joergen]

Originally posted by Infiniterb

The source is Z:\JurassicPark
The Destination is Z: DVDWRKDIR



You forgot to add a \ after Z:\DVDWRKDIR?

edit: and make sure you have latest avisynth installed and the mpeg2decdg.dll copied to its plugins dir (or the same dir specified insider dvd-rb if you used it).

[quantum]

RB Farm troubleshooting 101:

- Ensure you can encode correctly with DVD Rebuilder on the main PC.
- Ensure you can encode correctly with RB Farm on the main PC.
- Start, then abort an encode using DVB Rebuilder on the main PC. On secondary PCs, ensure you can open item.ecl and encode using eclcce.

If you can do these things, you should have no problems.

[b1ackmai1er]

Hi Quantum,

I am just about to download 1.1 and give that a try.

What I did find this morning is that if I try to load a farm.ecl file manually using CCE started with ECLCCE then I would get the same error as described previously.

However if I modified the farm.ecl file by putting the comments (;;) from REBUILDER.ECL at the top of file then it would load without error.

Regards b1m1

[b1ackmai1er]

Hi Quantum,

Just confirming that RB farm 1.1 works fine with CCE 2.50 (+ECLCCE1.81).

This is a dream come true.

Thanks b1m1

[acido]

@quantum
Thanks a lot for this tool, it worked really good for me cutting encoding time by half (2 pc used).
Hope it will be integrated in dvd-rb !

[Alex Z]

Originally posted by quantum
I completed a very rudimentary prototype. It works but it's too basic to post and I expect problems so I'd like to get an email correspondence going with testers to avoid cluttering up jdobbs forum.

@EddieTH I sent you a copy via email.
@Joergen I sent you a PM with the location and details.

Anyone else interested in testing this PM me (preferrably with an email address) and I'll send you a copy.

@quantum
Can psexec.exe (or simular) be used in future version of RB farm?
So you can start and control remote process remotely from one PC running RB farm app.

az

[EddieTH]

@quantum
Is there any chance for a version with CCE Basic support?

[Joergen]

I would recommend a program called RAdmin. It loads at startup, is very fast and low-cpu usage and you can control the other PC fully.
Far superior to software like PCAnywhere or WinVNC.

[Alex Z]

Originally posted by Joergen
I would recommend a program called RAdmin. It loads at startup, is very fast and low-cpu usage and you can control the other PC fully.
Far superior to software like PCAnywhere or WinVNC.


psexec is not your typical remote program. Its purpose is to execute a task (process) on remote PC (interactively or not). To make it even more simple to understand you can start calc.exe on PC 'B' using PC 'A'. In this scenario, RB farm will send command to execute ecllcce process on \\xyz pc wait for it to finish, execute next.

az

[quantum]

Originally posted by EddieTH
Is there any chance for a version with CCE Basic support? I would add support, but I only have the CCE Basic trial version which apparently has a different ECL format than the full version, so I can't do any testing.

[gunga]

Hello Quantum,

It seem everytime I click encode in RBFarm 1.1, I get this error "GetFileAttributes()Failed" on my networked computers. It works fine on the source computer, but not on any networked. I can map to any machine on the network and see all folders. I'm using CCE 2.67.00.23/eclcce 1.8b and DVDRebuild 0.44 on all machines. Any ideas as to why this is happening?

Thanks...

[EddieTH]

Originally posted by quantum
I would add support, but I only have the CCE Basic trial version which apparently has a different ECL format than the full version, so I can't do any testing.
If you decide to try it, let me know. I have the full version to test with.

[quantum]

I updated the link on page 1 to version 1.2. This version might work with CCE Basic, but it's untested. It definitely does not work with CCE Basic Trial which does not have ECL support.

If you want to try CCE Basic, you must type the filename manually on the "Path to ECLCCE" line in RB Farm.

Please let me know if it works.

[EddieTH]

@quantum
I'm in the middle of an encode right now, but I'll test it on my next project (later tonight) and let you know. Thanks.

[quantum]

Updated to version 1.3.

Added a new feature "Run Automatically"

The "Run Automatically" check box will cause the current instance of RB Farm to periodically detect if an encode has been started in the d2vavs directory. If a job has been started, RB Farm will automatically join in and process the next available chunk. You must start an encode on one PC by pressing the "Encode" button before any other instance of RB Farm will begin encoding automatically.

With this you could leave RB Farm running on PC's designated for encoding and not have to manually start the encode process on each PC.

[Joergen]

Originally posted by quantum
Updated to version 1.3.

Added a new feature "Run Automatically"

Ooh :eek: A true hard-core RBFarming option :D

[b1ackmai1er]

I am less than thrilled with the performance of my 2 PC farm.

HOST - CELERON 950, 100Mhz BUS, 768 Meg RAM = CCE SPEED .550
FARM - CELERON 600, 66Mhz BUS, 384 Meg RAM = CCE SPEED .150

I was wondering if others with low end machines could post some
speeds for comparison.

RB Farm is such a great tool but I guess I might just have to upgrade after all.

Also has anyone noticed that CCE encoding dies when a shared directory (other than those in use) is unshared?

Regards b1m1

[Joergen]

Originally posted by b1ackmai1er
I am less than thrilled with the performance of my 2 PC farm.


Definetly something wrong with your PC#2 or your PC#1 needs to put CCE on IDLE USAGE through ECLCCE.

My third PC is a Celeron2 850mhz 100fsb (oced) and does 0.46 when an XP2100+ does 1.20 and A64 3200+ does 2.10 (on the same project).

[EddieTH]

I can verify that RBFarm works with CCE Basic. I have 2 machines encoding with it right now.

Edit: I should mention that I'm starting RBFarm at Idle priority so it will start CCE at that priority as well. I put this batch file in the RBFarm directory (OldMacdonald.bat):

start /low RBFarm.exe


@Quantum: Thanks for the help getting it set up right. It's been so long since I've used any of my "serious" computer skills I've started forgetting the basics.

[puddy]

Anybody else getting oversized output with RB Farm?

If I encode on a singer computer using only
DVD-RB I get a "perfect" 4.32 GB disc.

Using RB Farm I got a 5.00 GB disc. All settings
were the same. Farm consisted of 3 computers (two
P4's and a P3). The RB Farm encode appears to be
fine in all other aspects, it is just too large!

Here's my stats:

Disc: Brother Bear Disc 2 NTSC
DVD Decrypter v3.2.1.0: Ripped in File Mode

DVD-RB v.45
- Removed: English DTS and AC3 French using DVD-RB
- Dynamically Assign Cell Bitrates Enabled
- Half-D1 and Half Space for Extra Enabled
- ConvertToYUY2() Enabled
- AudioDub(BlankClip()) Enabled
- CCE SP (+2.66+) Mode
- VBR_Bias=25
- Quality_prec=16
- VBR_Passes=3
- Add MPEG2DEC3DG to AVS File Enabled

RB Farm v1.3
CCE SP v2.67.00.23
eclcce v1.81
AviSynth v2.54 (all other versions uninstalled & registry cleaned)
MPEG2Dec3dg.dll v1.0.1.0 (put in AviSynth plugins dir)

Rb Farm is just very, very cool. Hope to be using it a lot in the future.

Ideas? :)

puddy

[quantum]

Before you ran the "rebuild" phase after the encode did you remember to deselect audio streams? Your deselections are not retained between starts and stops unless you save/load a project.

[wgw]

Originally posted by Joergen
Definetly something wrong with your PC#2 or your PC#1 needs to put CCE on IDLE USAGE through ECLCCE.

My third PC is a Celeron2 850mhz 100fsb (oced) and does 0.46 when an XP2100+ does 1.20 and A64 3200+ does 2.10 (on the same project).

b1ackmai1er, I'd say Joergen has the right idea. Set CCE on PC#1 to low priority. My dual MP2000+ does 2.2 while my second PC with a 1ghz Athlon does 0.9. However, if I unrar an archive on PC#2, CCE nearly stops processing completely even when both CCE and Winrar are both set to low priority. PC#1 will also drop to 0.45 if I am doing a lot of processing while CCE is running in the backround. CCE seems to be one app that could really use dual CPU's unless you can leave your computers alone for 6 hours or more while it is processing.

[puddy]

quantum: That is probably it! I'll test now.

puddy

[puddy]

That was it... I just had to go back and
uncheck the audio tracks and rebuild.

Thanks so much. :)

puddy

[quantum]

Care to share your numbers with us? The time to encode on one PC and the time for 3, and the processors?

[puddy]

1. P4 2.8 GHz, 800 MHz Bus, 200 MHz FSB, 512 MBs DDR-SDRAM PC3200
2. P4 2.66 GHz, 533 MHz Bus, 133 MHz FSB, 1 GB DDR-SDRAM PC2700
3. P3 600 MHz, 100 Mhz Bus, 100 MHz FBS (Laptop), 288 MBs 100MHz SDRAM

- No overclocking on any machine
- Linksys 100 Mbit switch with Cat-5e & Cat-6 wires

On the Brother Bear Disc 2 DVD (see previous post) the last CPU (the P3) finished up after 106 mins and only did 5 of the 76 cells. Plus 6 mins Prepare time and about 10 mins Rebuild time. So, a total of about two hours.

Machine #1 did the same disc in a little over 4 hours (240 min) standalone using one-click mode.

In both cases the source and destination were on different physical drives. But in the RB Farm run the source and destination were through mapped drives, even on the master CPU.

I noticed that when I ran the standalone test the Prepare process for this DVD was about 4 min and it was about double that through the mapped drives in preperation for the Rb Farm run. So, there is overhead that wouldn't normally need to be there. Maybe that is something could be looked at more?

So, that is just about what you'd expect from a setup like this, which is really cool.

I bet you'd be able to track some stats automatiaclly with a little coding that we could cut & paste back here for reference. That would be great! :)

Thanks again!

puddy

[puddy]

RB Farm Stats:
Disc: Dickie Roberts 16x9 NTSC (7.27 GBs Originally)
Removed: English 2-CH AC3 and French AC3 using DVD-RB
Prepare phase: 9 min
75 chunks total, VBR_passes=6
Machine #1: Processed 40 chunks in 235 mins
Machine #2: Processed 31 chunks in 228 mins
Machine #3: Processed 4 chunks in 229 mins

DVD-RB failed upon rebuild at segment 8 VOBID:6 CELLID:1 with the Run-time error '9': Subscript out of range error.
This same error happens with DVD-RB standalone in one-click mode as well. :(

My puddyFarm Stats:

HARDWARE
1. P4 2.8 GHz w/HT, 800 MHz Bus, 200 MHz FSB, 512 MBs DDR-SDRAM PC3200
2. P4 2.66 GHz, 533 MHz Bus, 133 MHz FSB, 1 GB DDR-SDRAM PC2700
3. P3 600 MHz, 100 Mhz Bus, 100 MHz FBS (Laptop), 288 MBs 100MHz SDRAM

No overclocking on any machine
Linksys WRT54G 100 Mbit switch over Cat-5e & Cat-6 wire (not using WI-FI in the puddyFarm)

SOFTWARE
DVD Decrypter v3.2.1.0: Rip in File Mode
DVD-RB v.45
- Dynamically Assign Cell Bitrates - ENABLED
- Half-D1 and Half Space for Extra - ENABLED
- ConvertToYUY2() - ENABLED
- AudioDub(BlankClip()) - ENABLED
- Add MPEG2DEC3DG to AVS File - ENABLED
- Run Encoder(s) Minimized - ENABLED
- CCE SP (v2.66+) Mode
- VBR_Bias=25
- Quality_prec=16
- VBR_Passes=6

RB Farm v1.3
eclcce v1.81
CCE SP v2.67.00.23
- Remove Cancel Button - ENABLED
- Priority set to IDLE

AviSynth v2.54 (all other versions uninstalled & registry cleaned)
MPEG2Dec3dg.dll v1.0.1.0 (put in AviSynth plugins dir)

Source and destination were on different physical drives on machine #1.
All machines, including the main, used two separate mapped drives for source and destination.


puddy

[EddieTH]

Since I tweaked some memory settings on my laptop (to compensate for low memory) and stopped using the onboard NIC on my main PC I've been getting between 1.5 and 1.9 on my main PC (2.4GHz P4/768MB), half that on my laptop (Celeron 1.7GHz/128 MB), and a third that on my wife's PC (Athlon 1GHz/256MB).

I'm getting the wife a new PC for mother's day, so I'll have another fast PC to use :cool: (when my wife's at work anyway). I'm considering getting a good NIC for the PC she's using now and just using it to serve the files.

[b1ackmai1er]

@wgw

Thanks for posting your celeron 850 stats. This confirms for me that my HOST PC is running fine. My FARM pc is running better now at around .250 . I think the movie I was encoding as a test (Lizzie McGuire) was slipping me a red herring, encoding speed was very low at the start of the movie but picked up speed later on. I suspect the animated character at the start of the movie was causing avisynth to load the decomb filter but I don't know how to confirm that.

@eddieTH

Wow! You run CCE with 128Meg of RAM! That's extreme!

@Quantum

Can I suggest you modify the "Run automatically" feature to do it's check every minute rather than every second.
Also can I suggest adding a "minimize on startup" so I can put it in my startup folder.


@puddy

Do you remember what you Machine 3 P3 600 CCE speed was?

@Joergen

Thanks for you suggestion. I tried various priorities on both my host and farm PC. Basically I found that High priority on the HOST killed the farm machine. Hight Priority on the farm machine made little difference but made the machine unusable for use. Low priority on both machines slowed both down a little. I found normal priority on both machines was about optimum.

[quantum]

Originally posted by b1ackmai1er
Can I suggest you modify the "Run automatically" feature to do it's check every minute rather than every second.It should be checking every 20 seconds. Too quick?
Also can I suggest adding a "minimize on startup" so I can put it in my startup folder.Put a shortcut in your startup.
In shortcut properties, change run->normal window to run->minimized

[puddy]

Originally posted by b1ackmai1er
Do you remember what you Machine 3 P3 600 CCE speed was?Something like .41 I think on the P3 600.

Would be cool of RB Farm would create a small log file of these stats that we could cut and paste into the forum for comparisons. :)

puddy

[puddy]

Originally posted by quantum
Put a shortcut in your startup.
In shortcut properties, change run->normal window to run->minimized Minimize to tray would be awesome. :)

[EddieTH]

Originally posted by b1ackmai1er

@eddieTH

Wow! You run CCE with 128Meg of RAM! That's extreme!


I found a registry hack that gave CCE a major speed boost. I started out with a speed of .30 to .35, but after the registry hack I get .70 to .95. Once I get my income tax refund (within a week) I'm going to be using the laptop as a laptop again (needs a new LCD) and I'll have a better PC to replace it with.

[quantum]

Can we have more details on the registry hack?

[EddieTH]

Oh yeah, forgot to include the details. :confused:

Look for the registry key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\PrefetchParameters\EnablePrefetcher

Set the value to 0 (Disable). That disables prefetch. CCE Starts a little slower but it performs a lot better. As with anything, YMMV.

Edit: I found this (don't remember where) listed only as a Windows XP hack, but that's the only Windows OS I'm running, so I haven't tried it on anything else.

[quantum]

Originally posted by Joergen
I would recommend a program called RAdmin. It loads at startup, is very fast and low-cpu usage and you can control the other PC fully.
Far superior to software like PCAnywhere or WinVNC. I use RAdmin and it works well. I just started using Remote Desktop which is part of XP. For some reason I never tried it before. I was very surprised at the great performance. Great combination of low CPU and high speed. Better than RAdmin IMO. It requires XP for the server part, but apparently the client part can run on other versions of windows.

[EddieTH]

I use Remote Desktop to run all my PCs for encoding and it works great. It's really just a limited version of Terminal Services. It's also nice for keeping your computer busy while you're at work. Of course I work from home so that's not really an issue for me.

[b1ackmai1er]

Hi Quantum,

Thanks for your feedback. I thought I could do something like that by creating a .PIF file or shortcut. But can't test it till my encode has finished.

How about an option to start encoding on startup? Or will "run automatically" do that even if the encoding process hasn't been started yet.

The reason I mentioned the checking time period was because I was looking at the network bandwith and saw packets heading out every second so I presumed RbFarm was doing a check every second. That made me think about the disk thrashing that would occur as every farm machine checked every second. Maybe the data is cached and it doesn't make any difference. Possibly I was mistaken and another application was accessing the network.

For one click integration with rebuilder is in not possible to set the Rebuilder CCE Application path to point to RBFarm or will rebuilder continue on to final step after the first RBFarm CCE encode is finished? If so maybe RB could add your completion check into Rebuilder. Is that as easy as it sounds?

I can't sleep. This is exciting stuff. Now to try that reg patch!

Regards b1m1

[SAPSTAR]

Originally posted by EddieTH
Oh yeah, forgot to include the details. :confused:

Look for the registry key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\PrefetchParameters\EnablePrefetcher

Set the value to 0 (Disable). That disables prefetch. CCE Starts a little slower but it performs a lot better. As with anything, YMMV.

Edit: I found this (don't remember where) listed only as a Windows XP hack, but that's the only Windows OS I'm running, so I haven't tried it on anything else.

Well...I tried on two boxes, a PIII 1000Mhz and a Celeron 1.7Ghz...NO change ?!?!? Maybe on the slowest computers it's more efficient....

[quantum]

I didn't see any change either on my Athlon 1.4.

[b1ackmai1er]

You could try this utility:

http://www.jester2k.pwp.blueyonder.co.uk/downloads/Windows-XP-Prefetch-Clean-And-Control.exe

Regard b1m1

[EddieTH]

Originally posted by SAPSTAR
Well...I tried on two boxes, a PIII 1000Mhz and a Celeron 1.7Ghz...NO change ?!?!? Maybe on the slowest computers it's more efficient....
I couldn't remember the specifics of why I did this, so I did some more research. It's specifically for low memory machines, so the speed of the CPU isn't nearly as important here. I only used it on the laptop because the others have decent memory.

[puddy]

RB Farm Stats for this run:
Disc: Italian Job NTSC (7.64 GBs originally)
Removed: English 2ch AC3 and French 6ch AC3 using DVD-RB
Prepare phase: 11 min
136 chunks total, VBR_passes=6
Machine #1: Processed 80 chunks in 239 mins (Server)
Machine #2: Processed 56 chunks in 245 mins (Remote)

DVD-RB failed upon rebuild at segment 2 VOBID:3 CELLID:1 with the "Run-time error '9': Subscript out of range" error. This same error happens with DVD-RB standalone in one-click mode as well. :(

My puddyFarm Stats for this run:

HARDWARE
1. P4 2.8 GHz w/HT, 800 MHz Bus, 200 MHz FSB, 512 MBs DDR-SDRAM PC3200 (Server)
2. P4 2.66 GHz, 533 MHz Bus, 133 MHz FSB, 1 GB DDR-SDRAM PC2700 (Remote)

No overclocking on any machine
Linksys WRT54G 100 Mbit switch over Cat-5e & Cat-6 wire (not using WI-FI in the puddyFarm)

SOFTWARE
DVD Decrypter v3.2.1.0: Rip in File Mode
DVD-RB v.45
- Dynamically Assign Cell Bitrates - ENABLED
- Half-D1 and Half Space for Extra - ENABLED
- ConvertToYUY2() - ENABLED
- AudioDub(BlankClip()) - ENABLED
- Add MPEG2DEC3DG to AVS File - ENABLED
- Run Encoder(s) Minimized - ENABLED
- CCE SP (v2.66+) Mode
- VBR_Bias=25
- Quality_prec=16
- VBR_Passes=6

RB Farm v1.3
eclcce v1.81
CCE SP v2.67.00.23
- Remove Cancel Button - ENABLED
- Priority set to IDLE

AviSynth v2.54 (all other versions uninstalled & registry cleaned)
MPEG2Dec3dg.dll v1.0.1.0 (put in AviSynth plugins dir)

Source and destination were on different physical drives on machine #1.
All machines, including the main, used two separate mapped drives for source and destination.

puddy

[Joergen]

It's the half-d1 option that causes the error 9 for alot of people now I think.

[puddy]

Originally posted by Joergen
It's the half-d1 option that causes the error 9 for alot of people now I think. I think you are right, but I like to point it out so that jdobbs can have the largest base of info to work with in tracking down the bugs. He's doing a great job already...

puddy

[quantum]

Updated the program to 1.4 and added a feature (from the readme)

REBUILD AUTOMATICALLY
The "Rebuild Automatically" check box will cause the current instance of RB Farm to attempt to rebuild the project after encoding using DVD Rebuilder. You should only select this option on the PC where DVD Rebuilder is installed and where you ran the prepare phase. DVD Rebuilder should be running with the settings for the current project correctly chosen, i.e., audio streams deselected, etc. The mode should not be one click mode, and "suppress warning prompts" should be selected.

I expect monster bugs with this. It's never easy remote-controlling another program. But if it works, this should further reduce the number of steps necessary to run a farm.

Theoretically, this could be the workflow:
1 - Run prepare with DVD Rebuilder
2 - Press encode with RB Farm (other PC's join in automatically, rebuild is done automatically)
Finished

[DDogg]

I'm refreshing like mad but the main link is still 1.2 and the alternate link is 1.3 for me. Suggestions? (yeah you guys are geeks, but damn smart geeks :) ) /Edit-Add/ Disregard, it came up. Nice job. I particularly appreciate the new lazy geek features. Heck, don't even have to get out to the chair now. (not tested yet)