effective MIRROR procedure?

[babaz]

hi you all,
here babaz from italian project

i'm pretty sure this issue has popped up before, but since i've not yet found anything *really* effective i'll put it down again

like other projects, i need to have an offline mirror of www.doom9.org as in sync as possible, thus i need to resync quite often
i've tried a lot of offline mirroring solutions out there (including the Scipio's way...) but never came up with something really effective by matter of time and data tranfered, i'll explain a bit:

as of now, i'm using WGET.exe (a GPL mirroring prog), i tweaked it a bit with a simple .bat procedure trying to achieve the most effective solution: this would be limiting the download of the *really* changed/new things (files, that is) from www.doom9.org, not everything again or something

but i'm pretty sure wget keeps downloading also unchanged items, maybe because they just have a different date-stamp...

so, any clue about this issue, translators? a hint from doom9 himself would be appreciated, since our efforts are mainly driven by how he sets up original site, that is

tnx 4 ur time, and keep on the outstanding work u all ;)

[Doom9]

I use lftp (also an open source app) to keep my local copy and the servers in synch. It really only uploads new files and it can be made to keep timestamps if I'm not mistaken. I'm using it in reverse mirror mode (thus it pushes files from my pc to the master server, and from there to all active mirrors) but lftp is mostly used in the reverse function which is just what you'd need. However, lftp only works via FTP. You do get the synch logs sent to you, right? So you can compare the logs you get by mail by what wget does to find out if wget is really trying to cheat you. There's one angle here that you may not have considered.. if you run wget on www.doom9.org you will not necessarily land on the same server so the file dates can be different (not every server is synched at the same speed and they are located in different timezones).

[Scipio]

ciao babaz :D

(not every server is synched at the same speed and they are located in different timezones).

a solution for this would be to use a specific IP of one of Doom9's servers. e.g.: http://65.43.78.94/

as for myself, I have to admit that I actually don't download a synch copy of Doom9.org anymore (and I haven't done that for a long time now).

I don't know why you would really need that most-up-to-date version of all files. When I want to change/update a certain file, I visit Doom9.org and download it right then - or else I would have to synch my local copy every day. I don't know why this would make sense?


and as for the software: Offline Explorer from metaproducts is very powerful but not freeware. It has plenty of options/filters for modified files and whether to download them or not.

I added a screenshot to illustrate some settings...

[babaz]

tnx u guys for ur prompt reply
i'll give a shot at lftp, as doom9 said, but anyway is it a win32 app??? i'll do some search, don't worry!

i didn't know www.doom9.org is actually running (mirrored) off more than one server... so maybe it's why i'm getting such inconsistent results, after all...

@doom9
how many mirrors do u have?

[Scipio]

plenty of useful info about doom9.org and its mirrors is available here:

http://forum.doom9.org/showthread.php?s=&threadid=42282


you could ask TheSwede for the win32 version of lftp since all google searches lead to the linux/unix versions of this GPL app. Or perhaps doom9 could send it to you (and to me too, if he doesn't mind). :D

[Doom9]

well... lftp won't do you any good unless you have ftp access...
lftp is a *nix app but there are w32 compiles.

[Scipio]

Originally posted by Doom9
well... lftp won't do you any good unless you have ftp access...
lftp is a *nix app but there are w32 compiles.


sure, but it would be still interesting for me since my new hosting has two servers and I could also use them the "round-robin-way"...
if you have the win32 binaries, you could graciously hand them over. :)

[DaveEL]

Originally posted by Scipio
sure, but it would be still interesting for me since my new hosting has two servers and I could also use them the "round-robin-way"...
if you have the win32 binaries, you could graciously hand them over. :)

My build is here http://www.ecs.soton.ac.uk/~del100/lftp.zip.

DaveEL

[Scipio]

thanks for your reply, DaveEL. (I got lftp from Doom9 already, but that will help others too).

http://www.ecs.soton.ac.uk/~del100/lftp.zip <- without the point at the end of the line, it works better. ;) (when clicking on your link you get a 404)

[Doom9]

blame vbb for that. You write a sentence ending with an url. Of course you put the dot just after the last letter since that's how you write a sentence.. but vbb turns that into a link having a dot at the end. One day I'll have to get that fixed.