RB Farm v1.7 - server farm for faster DVD Rebuilder encodes

[quantum]

Originally posted by DDogg
I particularly appreciate the new lazy geek features. Heck, don't even have to get out to the chair now. I guess I should finally add the ability to automatically start encoding without having to press the encode button, which I could probably do. Then it would be back to 1 click, prepare, and that's it :)

[EddieTH]

An update on my setup and what kind of speed I get from it. I found somebody to get another 128MB RAM from for my laptop, so it now has 256MB and I changed the prefetch registry setting to 1 (prefetch only for applications). My speed with that PC has gone up about 20%.

Source: Local hard drive in firewire external enclosure
Drive lettter Z: (mapped to Z: on remote PCs)

Working: Folder from internal hard drive mapped to W: on all PCs

*PC1: 1.50-2.15 P4 2.4GHz / 768MB RAM / Linksys NIC
PC2: 0.85-1.15 (laptop) Celeron1.7GHZ / 256MB RAM / Onboard 3Com 920 NIC
PC3: 0.50-0.85 Athlon 1GHz / 256MB (SD)RAM / Linksys NIC
*When encoding by itself, my main PC runs at 1.7-2.3.

Switch: Actiontec POS (okay technically it's an Actiontec 1524) DSL Router/Switch

Right now my biggest problem is having them all connected through a poor switch. I've never had a reason to use the network this way in the past, so it was never an issue, but since I started experimenting with networked encoding I've decided to upgrade soon.

[Mutant_Fruit]

I'm not sure if this has been mentioned already or not, but here's why bandwidth usage is so high for the outgoing connections from the server to the clients.

As far as i know, AVISynth does what it says on the tin, its synthesis an AVI file. It does this by on-the-fly decoding of the mpeg into an uncompressed format, which it then feeds to the different applications who open the .avs file. Thats why bandwidth usage is so high.

[Joergen]

Oh yes, good point. Also I wonder if the new mpeg2dec version with improved random access helps in this.

edit: no wait, avisynth runs on the reading PC and the conversion from mpeg2dec->avisynth avi is done in the RAM of the reading PC, not the server.

[EddieTH]

I just finished Matrix Revolutions (movie only from DVD Shrink) in 87 minutes encoding time and 19 minutes preparation and rebuilding. I noticed that CCE's speed on my main PC kept going down during the course of the encode process. When the other 2 machines stopped encoding their last segments and it got to encode alone again it shot from 1.29 to 2.11. I'll have to investigate this further, but I'm still happy with the time.

@quantum: I had the Rebuild Automatically option checked, and nothing happened. I also had DVD-RB running with all the correct settings.

[quantum]

Originally posted by EddieTH
@quantum: I had the Rebuild Automatically option checked, and nothing happened. I also had DVD-RB running with all the correct settings. Did you have the path to DVD Rebuilder entered in RB Farm? Did you have rebuild checked on the same PC you pressed the encode button? The first thing that should have happened is it should have shut down DVD Rebuilder automatically before it started encoding. If that didn't happen, you know it's not working.

It should also create a file farmproject.rbd in the rebuilder directory which holds the current settings.

[EddieTH]

Originally posted by quantum
Did you have the path to DVD Rebuilder entered in RB Farm?
Yes
Did you have rebuild checked on the same PC you pressed the encode button?
Yes
The first thing that should have happened is it should have shut down DVD Rebuilder automatically before it started encoding. If that didn't happen, you know it's not working.
Okay, I can say for sure that DVD-RB didn't close, but I just started another RBFarm encode with Auto Rebuild enabled and this time it did shut down DVD-RB. I'll let you know for sure in the morning.

[EddieTH]

@quantum: I didn't get my movie started last night due to network problems. This morning I got everything working again and tried the Automatic Rebuild option again. When I hit encode, nothing happened for a couple of seconds, then a new DVD-RB window opened (I still had the window open from the Prepare step) and I got this error:
Couldn't find DVD Rebuilder window at rebuild time.

Edit: Never mind. I shouldn't do this before I've had enough coffee. I forgot to change the working directory in RBFarm on my main PC so it was trying to rebuild a movie I finished yesterday. When I fixed the directory issue it seemed to work. Unfortunately, as I'm typing this, my network is #@!*# up again. I can't wait to get a new switch (next week hopefully).

[quantum]

Actually there were a few problems and it probably wouldn't have worked it most cases.

I fixed the major problems and it should be more robust now. You can test it this way:

If the d2vavs\farm_status directory, ensure farm_finished and farm_rebuilt are deleted.

Ensure farm_chunk_startedXXXX (choose the last number) is deleted.

Then start RB Farm and with the "rebuild automatically" option selected, press Encode.

[Joergen]

Btw since DVD-RB doesnt remember the audio you unticked, doesnt automatic rebuild mess that up?

[quantum]

Good question. The first thing that happens when you have auto rebuild enabled is RB Farm sends keystrokes to DVD Rebuilder to save the current settings (including audio deselections) in farmproject.rbd. Then when the encode is complete, DVD Rebuilder is launched and farmproject.rbd is reloaded, and the rebuild is initiated.

[Joergen]

:confused: :eek: :cool:

Wow! But be careful not to make RB Farm auto-migrating, or it will take over an entire network of computers, become self-aware and launch the nukes at 18:18.

[Joergen]

Hmm I just made an observation:

When rebuilding with mapped drives (O: and P: ) the rebuild time was 42 minutes.

When rebuilding with real drives (D: and E: ) the rebuild time was 21 minutes.

Under WinXP 1GB RAM and DMA enabled etc.

:helpful:

[quantum]

Prepare and rebuild are disk intensive. The best performance should be achieved by doing these on the serving computer where the files are local.

[Joergen]

Yes thats what I was saying (I can see I couldhave explained more).

It was the main computer but I rebuilt with the mapped paths (so D: was mapped to O: and E: to P: ). I can understand microsoft didnt intend somebody to use the network mapping feature for cloning their discs but clearly the mapping makes the drive access times twice as long.

And D: is HD#1 (7200rpm maxtor) and E: is HD#2 (another 7200rpm maxtor), both NTFS4 (or whatever version winxp likes to do it with).

Feel free to try it on your own. :)

[quantum]

Now I understand. Maybe you can change the real drive letters to O and P using disk manager and map drives D and E if you still need them for your other stuff. Or maybe there is a more efficient way to clone a drive letter.

[Joergen]

I dont want to reverse the mapping cause then my games would load slower when I play ;)

But its no biggie to change the paths before rebuilding, just a few clicks.

But if somebody else is wondering why the rebuild takes longer than usual.

[quantum]

Try the subst command. At the cmd prompt, type:

subst o: d:\

o being the new drive and d:\ being the path

[Joergen]

OMG I havent seen that command for years! Thanks for reminding me.
I'm rebuilding now with substituted paths for both drives and the speed seems consistant to the 21minute rebuild with real paths.

edit: Yep, subst paths gave me 22minutes. :)
I'll just add them to a batch file and copy it to the startup folder to map them after each reboot.

:helpful:

[quantum]

New update 1.6

Upgraded auto encode functionality. You no longer need to press the encode button on a single PC to start the encode process. This should allow one click functionality on a farm. You should be able to press "prepare" in DVD Rebuilder and RB Farm will automatically do the rest.

There isn't much more that can be done with this. The only thing that isn't being supported is batch jobs. I haven't heard any requests for that, so hopefully this will be the last update for a while, except for bug fixes.

[EddieTH]

I hadn't thought of subst. I've just been using a local Z drive that's mapped on the other PCs for my source VOBs and a mapped drive on all PCs for the working directory. Then I've been changing the working directory in DVD-RB to the local drive instead of the mapped drive before I rebuild and it runs at full speed that way. Using subst means I can be even lazier. Very cool.

@quantum: I agree. There isn't much more to be done with this. It does everything it was intended to do already. Thanks for a nice program.:)

[b1ackmai1er]

Would some kind soul be able to mirror the latest version of RBFarm for me? I have been unable to download any of the last three versions even after clearing my browser cache.

Thanks b1m1

[b1ackmai1er]

@Joergen

Did you try out the subst command? I found I couldn't share a subst'ed drive and was wondering what your milage was.

@quantum

Any chance you could program RBFarm to accept network paths instead of drive/folder paths. i.e. \\celeron950machine\d:\DVD-RIP. This would also require the folder paths in the ecl files to be updated before executing eclcce. ??

Also like to suggest Rb Farm start encoding the largest segments first so that we don't end up with a situation where encoding has finished except for the largest segment which encoding on the slowest machine.

Regards b1m1

[Joergen]

You cant use the subst on network drives, only local ones. But that was my purpose, to use it to create virtual drive letters for the main computer so I can run the prepare phase with the same paths as all the nework computers. But the subst way of drive mapping is way faster on the localhost level than the network neighborhood way.

edit: Yeah that largest segment first (or at least not last) is a good idea. That way if the worst segment is picked up by the slowest PC it has time to finish it.

For now I've usually hit END and cancel on the slowest PC if it picked up a huge segment just as the faster pc's are finished, and then just encode that last bit with a faster pc.

[smlong426]

I can encode just fine with RB-FARM, but when it finishes, I load Rebuilder and click 'Rebuild' I get an error saying "There is nothing to do!"

Any suggestions?

[Joergen]

Well maybe one of the computers isnt writing to the same D2AVS path? If theres even one .m2v missing DVD-RB will say theres nothing to do.

Also an old version of ECLCCE writes .MPV files instead of .M2V, so update to 1.8 or newer on all PCs.

[smlong426]

Originally posted by Joergen
Well maybe one of the computers isnt writing to the same D2AVS path? If theres even one .m2v missing DVD-RB will say theres nothing to do.

Also an old version of ECLCCE writes .MPV files instead of .M2V, so update to 1.8 or newer on all PCs.

Updating EclCCE made this problem go away. Thanks.

[quantum]

Originally posted by b1ackmai1er
Did you try out the subst command? I found I couldn't share a subst'ed drive and was wondering what your milage was.
You don't need to share the subst drive, you share the original drive. On the other computer, you assign the correct drive letter when you map. This assumes the drive letter where the dvd files are stored is taken on other computers. Otherwise this is a non-issue:

computer1:
- dvd stored on drive d:
- subst m: d:\
- share d: as moviedrive

computer2:
- drive d: is taken
- map \\computer1\moviedrive as m:

The path for DVD Rebuilder and RB Farm(s) would be m:\yourdirhere

[b1ackmai1er]

Thanks everyone.

[NobbyNobbs]

Originally posted by quantum
There isn't much more that can be done with this. The only thing that isn't being supported is batch jobs. I haven't heard any requests for that, so hopefully this will be the last update for a while, except for bug fixes.

I would like the possibility to use RBFarm for batch jobs, if I go to work leaving 3 computers with one DVD to do each, when I get home one will not be finished, one will just have finished a couple of hours ago, and the fastest will have finished several hours ago.
If I could use RBFarm for this, I could have finished 3 to 4 DVD´s.

I often use a long time with different tools (DVD Shrink, DVD ReMake, IfoEdit, etc.-) to prepare a disc for encoding, so it would be nice to have the computers finish of the work when I´m sleeping, or at work. :)

[wgw]

Originally posted by b1ackmai1er
Would some kind soul be able to mirror the latest version of RBFarm for me? I have been unable to download any of the last three versions even after clearing my browser cache.

Thanks b1m1

This happened to me once. Try manually deleting all temporary internet files in your cache folder including cookies.

[b1ackmai1er]

@Quantum

RbFarm is working really well now. I would appreciate batch jobs support as well.

Thanks for such a great program.

Regards b1m1

[b1ackmai1er]

@Quantum

I must have had a hiccup with my RB Farm encode today because I had to restart it. It finished ok but when I Rebuilt with Rebuilder I got an error. I went back into the rbfarm directory and found that one segment had not finished encoding.

There was a farm_chunk_started0035 but no farm_chunk_finished0035 file. However there was a farm_finished file.

I restarted RBFarm and expected it to pick up this incomplete farm chunk and restart it, but it didn't.

I deleted the farm_finished file and restarted again but it still didn't pick it up.

Finally I succeeded by deleting both the farm_finished file and the farm_chunk_finished0035 file.

I thought about this for a while and then realized that this makes sense because RbFarm has no way to know if a farm PC has crashed or is still running that incomplete chunk. But it does know there are no more unstarted chunks so it writes a farm_finished file

So I wanted to suggest that you add a message that notifies the user in such a scenario. ie when you start or shut RB farm it checks if there is a farm_finished file. If there isn't it continues on respectively. If there is it should then check if it has any farm_chunk_startedXXXX files that are missing corresponding farm_chunk_finishedXXXX files and alert the user.

Hope this is clear and not to pedantic.

regards b1m1

[quantum]

If an encode is improperly aborted, you could have stranded chunks that won't be picked up. You can fix it the manual way that you described. I could code something to deal with it, but I probably won't be doing any updates for a while.

[tomaste]

Originally posted by quantum
If an encode is improperly aborted, you could have stranded chunks that won't be picked up. You can fix it the manual way that you described. I could code something to deal with it, but I probably won't be doing any updates for a while.

Oh? I was hoping to try RB Farm this week. Why the "no updates" ?

[quantum]

Because I don't feel like it. Why does it matter? Does the idea the program won't be changed every day turn you off from trying it?

[winny]

Superb program, thanks for all your efforts to date Quantum.

RBFarm works a treat from main PC, but I have a small problem with the second machine. It only creates small vaf and m2v files, whereas the main PC processes the files with no problems.

I am using eclcce 1.81 and cce 2.67.0.23.

Has anyone seen this before, or have any suggestions for a possible cause?

[wink]

Originally posted by b1ackmai1er
@Quantum

I must have had a hiccup with my RB Farm encode today because I had to restart it. It finished ok but when I Rebuilt with Rebuilder I got an error. I went back into the rbfarm directory and found that one segment had not finished encoding.

There was a farm_chunk_started0035 but no farm_chunk_finished0035 file. However there was a farm_finished file.

I restarted RBFarm and expected it to pick up this incomplete farm chunk and restart it, but it didn't.

I deleted the farm_finished file and restarted again but it still didn't pick it up.

Finally I succeeded by deleting both the farm_finished file and the farm_chunk_finished0035 file.

I thought about this for a while and then realized that this makes sense because RbFarm has no way to know if a farm PC has crashed or is still running that incomplete chunk. But it does know there are no more unstarted chunks so it writes a farm_finished file

So I wanted to suggest that you add a message that notifies the user in such a scenario. ie when you start or shut RB farm it checks if there is a farm_finished file. If there isn't it continues on respectively. If there is it should then check if it has any farm_chunk_startedXXXX files that are missing corresponding farm_chunk_finishedXXXX files and alert the user.

Hope this is clear and not to pedantic.

regards b1m1

I second that

[NobbyNobbs]

Originally posted by winny
Superb program, thanks for all your efforts to date Quantum.

RBFarm works a treat from main PC, but I have a small problem with the second machine. It only creates small vaf and m2v files, whereas the main PC processes the files with no problems.

I am using eclcce 1.81 and cce 2.67.0.23.

Has anyone seen this before, or have any suggestions for a possible cause?

When I tried it with four machines, the same happened on one of them.
I had an old version of AVISynth on it, uninstalled all versions, reinstalled 2.54 and everything worked like it should.

[tomaste]

Originally posted by quantum
Because I don't feel like it. Why does it matter? Does the idea the program won't be changed every day turn you off from trying it?

No no, I was just afraid you were "stopping" development. When you said "I could code something to deal with it, but I probably won't be doing any updates for a while." It sounded like something had put you off, and you were shelving it. I didnt want to fall in love, just to have my heart broken. ;)

Everyone needs some downtime! Have a good break.

On a side note, I will be trying this with a farm machine that has 4 xeon processors. Any experience with something like this?

[Joergen]

I'm still using 1.0 cause I dont need the features of the new versions! Quantum went much further than I needed :)

[280zx]

Would a slow powerline network be adequate for rbfarm? The network can only manage just under 3mbps. My main pc is a 3.2ghz p4 and the secondary pc is a 2.66ghz p4. I've already ordered a patch cable to connect them directly, through a router.

[Joergen]

I'd say homepna etc is not fast enough for your PC speeds. I havent tried yet but I'd suspect 10mbit starts to be too slow aswell.

[Joergen]

No such problems here. I've done about 10 encodes with RBFarm so far.

Check that you have the latest avisynth and ECLCCE on both PC's, ie the same exact setup on both. And check the paths and that none of the .m2v chunks contain the avisynth errors.

[wink]

i am pretty sure it's the same setup. The fact that it runs perfectly fine with progressive parts. I believe the movie play fine, just undersize. I can also check again later tonight.
by itself, both computer encode interlace material just fine. It's just a problem with network rbfarm & interlace material for me
btw, movie is queen of the damned
i found the prob. my careless mistake.

[quantum]

Are there any P4 Hyperthreading RB Farm users? If so, report the percentage of CPU used by RB Farm in comparison to the CPU DVD Rebuilder takes on the local PC.

[280zx]

I finally got around to setting this up, and it's very cool. With a 3.2ghz p4 and a 2.6ghz p4, the network traffic averages out at just over 13 mbps but it jumps around pretty wildly with 30mbps spikes.

Originally posted by quantum
Are there any P4 Hyperthreading RB Farm users? If so, report the percentage of CPU used by RB Farm in comparison to the CPU DVD Rebuilder takes on the local PC.

I'm a little late with this info since the cpu usage has been fixed in dvdrebuilder, but it's about the same now.

Anyway, nice work quantum, thanks.


Edit- RBfarm seems to have overloaded my router, a wired 10/100 linksys. After the backup my internet speed was about 1/4 what it's supposed to be. I had to restart my router to correct the problem. Bittorrents used to do the same thing until I switched to the azureus client.

Edit 2 - It only seems to have crashed my router the first time I used it. The last 3 backups went perfectly. Cool :D

[Joergen]

Just thought I'd add a note.

The new MPEG2Dec3dg.dll that comes with the http://www.doom9.org/Soft21/Decoders/decodefix110.zip package really does improve random access (HD usage in general) especially for RB Farm.

I remember reading about it earlier but didnt bother to update. But today I was listening to the HD of the serving PC clunking wildly still after only one network PC was still encoding and I thought "does it really need to trash around so much for a few megabits".

Apparently with the new mpeg2dec, it now doesnt! :)

Or am I just dreaming?

[djackson]

I had RBFarm 1.1 working perfectly and upgraded to 1.6 and am having probs on the "Farmed" PC's.

All encoding works fine on the server. When loading REBUILDER.ECL on the Farms, all chunks are read as 240 frames.

On server:
U:\ is source material (local drive)
V:\ is target and D2VAVS material (local drive)
d:\dvd-rebuilder\rebuilder.exe - RBFarms RB dir
v:\dvd-rebuilder\thegame\d2vavs\ - RBFarms D2VAVS dir
d:\cce sp trial\eclcce.exe - RBFarms ECLCCE dir

On Farms:
U:\ is source material (network drive)
V:\ is target and D2VAVS material (network drive)
v:\dvd-rebuilder\thegame\d2vavs\ - RBFarms D2VAVS dir
d:\cce sp trial\eclcce.exe - RBFarms ECLCCE dir

Any advice appreciated. Considered returning to v1.1, but overwrote all copies.

[quantum]

That has to be a coincidental problem. RB Farm is just feeding ECL's to CCE. Try encoding on the farm PC's with DVD Rebuilder. If that doesn't work, you have a problem somewhere else such as avisynth.