@ all apache experts

[Doom9]

so I should use something like
RewriteEngine on
RewriteBase /~doom9/public_html/Soft21/
RewriteCond %{HTTP_REFERER} !^http://([^/]*\.)?doom9\.(org|net)/
RewriteRule \.(zip|rar|exe)$ http://www.doom9.org/leech.htm [R]

RewriteCond %{HTTP_REFERER} ^http://([^/]*\.)?doom9\.(org|net)/
RewriteRule ^/Soft21/(.*)$ http://anothersite.tld/somepath/$1 [R]

is that right?

[nek]

you don't even need the

RewriteCond %{HTTP_REFERER} ^http://([^/]*\.)?doom9\.(org|net)/

line,

it will implicitly be an "else" since the first rewrite rule won't be matched

hope this works

[Rasqual]

Here's a masqueraded URL:
 http://www%2emultimania%2ecom%2fskra%2f.doom9.org/stealer.html (copy and paste in your browser)
which translates to:
 http://www.multimania.com/skra/.doom9.org/stealer.html
The latter does pass, not the former.

I suggest putting directly the authorized vhosts:
RewriteCond %{HTTP_REFERER} !^http://((www|srvX|whatever)\.)?doom9\.(org|net)/

---
{Edit Note}
The .htaccess protected server is server3,
phpinfo is here to check out the referer (lazy to write a script ^_^; )

[tCS [hF]]

hey folks,

since allmost anything has been said already, my final tip is to enable debugging while writing regular expressions with mod_rewrite. to do this, include the following statement in your conf:

RewriteEngine on
RewriteLog "/www/logs/rewrite.log"
RewriteLogLevel 0

put the rewriteloglevel to something reasonable, say 3, then you'll see how your request is being rewritten. this saves a _lot_ of time while searching for possible bugs.

a look at some regular expression book say this one (http://www.amazon.com/exec/obidos/ASIN/1565922573/qid=101) might help, or at least the unix manpages regex(3) and regex(7).

HTH
tCS

[Doom9]

@rasqual: I don't get your point.. both these urls don't pass your referer check test with either regexp.. and the one we're using now:
!^http://([^/]*\.)?doom9\.(org|net)/

says either doom9.org/net, or *.doom9.org/net where the * must not include a / so it can't be a subdir on a server

[Doom9]

@nek: I'm afraid that won't do it. Here's what I have now:

RewriteEngine on
RewriteBase /~doom9/public_html/Soft21/
RewriteCond %{HTTP_REFERER} !^http://([^/]*\.)?doom9\.(org|net)/
RewriteRule \.(zip|rar|exe)$ http://www.doom9.org/leech.htm [R]

RewriteCond %{HTTP_REFERER} ^http://([^/]*\.)?doom9\.(org|net)/
RewriteRule ^/Soft21/(.*)$ ftp://ftp.doom9.org/$1 [R]

while it blocks unauthorized access, the download still comes from server2.doom9.org and not from ftp.doom9.org as it should. I tried leaving the rewritecond before the 2nd rewriterule away but that didn't help either.

So I'm back at the .htaccess we currently have:

RedirectPermanent /Soft21 ftp://ftp.doom9.org/

So.. how can I combine these two? I want the ftp redirect to only apply if access is allowed, but then it must be rewritten.

[Rasqual]

Originally posted by Doom9
@rasqual: I don't get your point.. both these urls don't pass your referer check test with either regexp.. and the one we're using now:
!^http://([^/]*\.)?doom9\.(org|net)/

says either doom9.org/net, or *.doom9.org/net where the * must not include a / so it can't be a subdir on a server

That's why the / has been masqueraded:
In http://www%2emultimania%2ecom%2fskra%2f.doom9.org/stealer.html,
the part www%2emultimania%2ecom%2fskra%2f. does match ([^/]*\.)? .
As you said, in the *.doom9.org pattern, * cannot have "/" that's why they're hidden, changed into %2f!

Please note that this method will _not_ work on Netscape 4.75 but has successfully passed in IE5 and IE6.

[Maz]

I'm no expert in mod_rewrite, but looking at their examples, it seems you should write :
RewriteRule ^/Soft21(.*)$ ftp://ftp.doom9.org$1 [R]

if this doesn't work, you should also try
RewriteRule ^/Soft21(.*) ftp://ftp.doom9.org$1 [R]

RewriteRule ^/Soft21(.*)$ ftp://ftp.doom9.org:21$1 [R]

(exactly ftp://ftp.doom9.org:21/pub/doom9$1, if I get it right)

On another hand, you should remove the trailing slash if there is one from the RedirectPermanent.

Hopes this helps

Maz

[Doom9]

maz: thanks for the suggestion but that can't be it either. IF it were a problem of pathnames the downloads would not work.. you'd get an url like ftp://ftp.doom9.org//... but.. there's no redirect taking place in the first place

[Lunatic]

:confused: I don't understand, why you think, other pages will link to your files at your HTTP-Server which you try to prevent, when they could link to them at your FTP-Server where they are not protected ... :confused:

[mat813]

well, they are not protected, but :
1)
try to browse ftp.doom9.org:/pub/doom9

2)
I host/admin the ftp server, and I don't really care for bandwidth

[Lunatic]

There is no need for browsing the directory at the ftp server. The Apache webserver would tell the browser (or an evil webmaster who's stealing links) the appropriate path/filename when doing the redirect.

I think it makes no sense to do a referer check, when you keep the ability to get the files directly by ftp. The referer check consumes cpu time while not doing anything useful, since only links by doom9 would be checked (because links from others would point to the ftp when they've once realized the links won't work over http).

[jr]

Pretty much only way to make sure others don't steal links to FTP server is use separate account and change password often. Then you just change ftp://ftp.doom9.org/file.zip to ftp://user:pass@ftp.doom9.org/file.zip.

Alternate option would be moving files around on FTP server. Eg. ftp://ftp.doom9.org/alsjdsk/file.zip and changing it to ftp://ftp.doom9.org/kksdlks/file.zip couple days later.

Either one will mean that any proxies between user and FTP server would be useless which means more traffic on FTP server.

[Lunatic]

I don't think renaming is a good idea ...

It's not too difficult to write a little script that gets the current URLs from doom9.org once a day. If the files are accessible somewhere without referer check (ftp), someone will find a way to link to them.

I think the best idea is to serve the files via http.

[Rasqual]

@Lunatic:
Your remark is correct, but what is actually more problematic is the quantity od used bw, not the CPU usage. As for the http redirect to the ftp, I also think the same way: it can easily be circumvened.

What would be better is to change the address dynamically every day (that would imply tight association between Webserver and FTP).
What would be even more interesting is some kind of Kleenex® login/pass, that could be used only once (and then made invalid) and which have a limited lifespan. Such systems have been seen on irc if I remember well, and I liked the idea. Another problem would then be raised: how to generate a login for each download, how to manage this from the FTP server's point of view (might be specific), etc.
I must admit I don't know how all this would be done.

Any suggestions?

[mat813]

well, opie ftp access would be possible, just a bit painful to put up, I've already though of such things I would not be to difficult with pam, but someone else would have to look into it because days only have 24 hours in here and I don't have time for it :)

[jr]

Generating temporary ftp accounts with CGI isn't that difficult if you have necessary rights on server. However I don't think Doom9 has enough rights to create FTP accounts so that approach won't work.

How about this:

1. User clicks link on webpage and is sent to http://server9.doom9.org/files/HugeDownload.zip

2. CGI on server9 checks referer and other headers

3. CGI creates directory under /files which is in 0111 mode to prohibit directory listing. [ eg. ftp://ftp.doom9.org/files/kakakakfjdk/ ]

4. CGI creates hardlink of HugeDownload.zip under /files/kakakakfjdk/ directory

5. CGI returns either 302 pointing to ftp://ftp.doom9.org/files/kakakakfjdk/HugeDownload.zip or generates webpage with ftp link. I'd prefer latter one since I often surf web with one PC and download files to my unix shell that has faster connections and then transfer files to home while sleeping..

6. Cleanup process removes hardlinks and folders every now and then. Perhaps every hour or two.


There's still same problem with someone creating script that generates download requests for files, goes thru CGI to get final FTP url and passes that URL to user as their own download site.

Even if someone would create script that generates one-time FTP accounts it's possible to write script that fetches list of those OTP FTP accounts and passes them forward.

Of course it's a easy to ban IP such script would use. Next thing malicious webmaster might do is using some free proxy list to come thru different IP each time..

Only way to make sure no-one steals your bandwidth is using HTTP downloads or making FTP link stealing hard enough so lame webmasters don't have time nor skills to defeat it. There's always someone that will break thru it but that's small minority, right?

[mat813]

Originally posted by jr
1. User clicks link on webpage and is sent to http://server9.doom9.org/files/HugeDownload.zip

2. CGI on server9 checks referer and other headers

3. CGI creates directory under /files which is in 0111 mode to prohibit directory listing. [ eg. ftp://ftp.doom9.org/files/kakakakfjdk/ ]

4. CGI creates hardlink of HugeDownload.zip under /files/kakakakfjdk/ directory

5. CGI returns either 302 pointing to ftp://ftp.doom9.org/files/kakakakfjdk/HugeDownload.zip or generates webpage with ftp link. I'd prefer latter one since I often surf web with one PC and download files to my unix shell that has faster connections and then transfer files to home while sleeping..

6. Cleanup process removes hardlinks and folders every now and then. Perhaps every hour or two.

looks like very long for me, here is a simpler approach :
1. user clicks on http://server9.doom9.org/files/HugeDownload.zip

2. CGI creates a one time account on the ftp and redirects the browser

3. when the client has logged onto the ftp, the account is automatically disabled

end of it :)

[Rasqual]

Originally posted by mat813


looks like very long for me, here is a simpler approach :
1. user clicks on http://server9.doom9.org/files/HugeDownload.zip

2. CGI creates a one time account on the ftp and redirects the browser

3. when the client has logged onto the ftp, the account is automatically disabled

end of it :)

Yup, and clean temp account after a given amount time has passed. Any stealer may write a tool to generate fake http requests just to get any number of ftp pass.

Now, what if a webmaster uses a script or program on his own server that would connect thru http, send a fake referer field (matching a correct doom9 of course) and redirect the user to the address Apache sent back? php with sockets might do, I think.

[cyberrobo]

hu!

2 theoretical solutions for your problem

a) some php scripts which set/read cookies on the client machine (data gets written on the client machine when visiting the news/download/whatever section, which gets compared when trying to download stuff from the server (also via another script)). afaik the download scripts included in phpnuke (or at least a hack for it) provide these functionality

b) get a ftpd which is capable of authenticating users via a (my)sql-database (like proftpd). this is needed because creating "real" system accounts on the fly is quite painful (slow) and somewhat insecure. the accounts should be created on the fly when the user retrieves the page with the download links, the password protection should be invisible to the (l)user (embedded in the download link like ftp://foo:bar@ftp.doom9.org/ficken.zip). these "temporary" accounts should get invalidated after a short time-period (30 mins?).

The first solution works well on some major sites which are similar to yours (providing free (semi-illegal) information for a "small" computer-related scene), the latter is only needed when you have to provide the information/content via ftp.

if you have further questions you can contact me at research (at) amd (dot) co (dot) at, because i don't think i've got the time to follow this thread/forum

mfg michael