@ all apache experts

[Doom9]

I tried to implement a referrer check via the following .htaccess file that I put into a software directory, then tried to download from the server itself, and just pasting the url into an empty browser.. and both worked :/

<IfModule mod_rewrite.c>
RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://www.doom9.org [NC]
RewriteCond %{HTTP_REFERER} !^http://doom9.org [NC]
RewriteRule .*\.zip|rar|exe$ http://www.doom9.org [R]
</IfModule>

I also tried to remove the IfModule condition to provoke an error should the module not be installed.. but it still was a nogo. Any ideas?

[c0p0n]

have U tried to put the path of doom9 on the server? not the http one, but the file one... that would restrict the download from a page of the same server but should work

[omol]

Originally posted by Doom9
I tried to implement a referrer check via the following .htaccess file that I put into a software directory, then tried to download from the server itself, and just pasting the url into an empty browser.. and both worked :/

Doom, is there any directive in the main httpd.conf enabling per-directory .htaccess parsing? Also, I'm not sure if the 1st rewrite condition test is a valid regexp. If a test of blank is intentional, better use "" instead.

regards,
omol

[Koepi]

Doom9, i wrote you some months back that trailing shlashes are important on URLs.

So the documentation for mod_rewrite says:
DON'T FORGET THE TRAILING SLASH.

put "http://www.doom9.org/" in there and it should work like expected.

(Btw., if you move the mouse over the www-button next to your posting - there you still forget about that slash as well :P

Best regards,
Koepi

[realSpiderman]

I think, it would be better to write the RegExps in the following way:

^http://www.doom9.org/.*$

And the substitution should IMHO look like

RewriteRule .*\.(zip|rar|exe)$ http://www.doom9.org [R]

An I think, the only way to really test, if mod_rewrite is installed, will be to use only a rewrite-rule without conditions, and see if it redirects.

[Doom9]

@c0p0n: care to give me an example?
@omol: what would that line look like?
@koepi: it doesn't exactly say that.. this clause is very well hidden and most of the examples don't have a trailing slash either. Furthermore even in my Mosaic times I did never have to use any trailing shlash.. and my browser adds a trailing slash automatically to my www button (and all the other urls for that matter).

does anybody has a working example that he actually tested?

I want every .zip, .rar and .exe file in the relative directory Soft21/ and its subdirectories to be checked for a doom9.org/net referrer and redirect to the index page if it doesn't match.

[xming]

You should really check the main config's per directory config.

somewhere you have to put

<Directory /youdir/htdocs>
AllowOverride All
<Limit GET POST>
Order allow,deny
Allow from all
</Limit>

[Koepi]

Hi again,

well, I can give you one example of my rewrite rules here:

(taken from my local file /users/koepi/www/.htaccess)

AuthUserFile /dev/null
AuthGroupFile /dev/null

<IfModule mod_rewrite.c>
RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://www.<someserver>/~koepi/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://<someserver>/~koepi/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.<someserver-alias>/~koepi/.*
$ [NC]
RewriteCond %{HTTP_REFERER} !^http://<someserver-alias>/~koepi/.*$ [N
C]
RewriteCond %{HTTP_REFERER} !^/~koepi/.*$ [NC]
RewriteRule .*\.(zip|gz)$ - [F]
</IfModule>

You can leave that <IfModule> directive away if mod_rewrite is built into the server - i just do this because I don't have much traffic so this doesn't affect the server load and makes sure that the site is still available even if i rebuild my apache without mod_rewrite ;) .

Btw., if you're concerned about traffic, then you'd better build your links with trailing slash. Of course it works without, but it does the following:

client to server:
gimme http://www.doom9.org
server to client:
(I only have http://www.doom9.org/)
client to server:
ok, gimme http://www.doom9.org/
server:
ok, here you are.

So it's one request more that could have been avoided.

Sorry for that very simplistic example, but it illustrates the easiest way what's going on. Agreed, it's not much traffic, but it _is_ traffic (unneccessary even).

That's what this whole thing with the URL is about. Sorry that I didn't make that clear before.

I hope this is helpful input.

Regards,
Koepi

[Doom9]

do these rules also work for your subdirectories or do you have to put them into every directory?

@xming: should that be in httpd.conf? do the first two lines of the script take care for every doc being served or does it have to be set up account specific (like
<Directory /home/doom9/public_html>
AllowOverride All
<Limit GET POST>
Order allow,deny
Allow from all
</Limit>
and be repeated for every account that has web access?

you can guess from all this that I don't have access to the server config file so things are a bit complicated

[Koepi]

I didn't check if this works for subdirectories as well since I've everything in the main www-dir there.

My guess is that you'd have to place such an .htaccess in every subdirectory that should be referrer checked...

in your case I would place it into .../Soft21/.htaccess

Regards,
Koepi

[omol]

Originally posted by Doom9
@omol: what would that line look like?

In httpd.conf, or maybe access.conf depends how the admin manage the www box, you will notice some "<Directory>" containers. Locate the one refers to your web's root dir. Make sure the "AllowOverride" directive IS NOT set to "none". I would not recommand using "AllowOverride ALL" in respect to the problem you are trying to solve, "AllowOverride FileInfo" is more appropriate and won't risk opening another can of worms. And yes, this directive is recursive, so you don't have to set "<Directory>" container to every directory within the path, nor you don't have to drop in .htaccess to every directory within the path. Also, the "AccessFileName" directive also worths checking, make sure it is set to ".htaccess".

@koepi: it doesn't exactly say that.. this clause is very well hidden and most of the examples don't have a trailing slash either. Furthermore even in my Mosaic times I did never have to use any trailing shlash.. and my browser adds a trailing slash automatically to my www button (and all the other urls for that matter).

I agree with koepi here. The trailing slash is formal and correct, and make perfect sense. Just try to telnet to port 80 of the www server and GET.....;)

regards,
omol

[realSpiderman]

@doom9:
1. Actually, it's rather easy to test, if mod_rewrite is installed and works. Just read my above posting again.

Try a rule without conditions. If not every GET in the directory is redirected, mod_rewrite is either not installed or .htaccess is disallowed. (That is the Default on most Apache-Distris.)

2. Your RegExps will not match anything useful, because you forgot the ".*$" at the end.

[c0p0n]

<IfModule mod_rewrite.c>
RewriteEngine on
RewriteCond %{HTTP_REFERER} ^$
RewriteCond %{HTTP_REFERER} ^http://www.doom9.org.* [OR]
RewriteCond %{HTTP_REFERER} ^http://doom9.org.* [OR]
RewriteCond %{PATH_INFO} ^/home/doom9/htdocs.* [OR]
RewriteRule ^/$ .*\.zip|rar|exe$ http://www.doom9.org [R]
</IfModule>

note that the "!" symbol says "abscence of". the line:

RewriteCond %{PATH_INFO} ^/home/doom9/htdocs.* [OR]
should be modified to the local directory of the html docs to use as referers...
maybe this could help

[Doom9]

can you guys agree on anything.. as in the thread where I first read about this operations I get 20 different opinions. the absolute path for my home on the server I'm testing is /usr/home/dom9/public_html/ (that's the root of the doom9 page, for the software stuff you'd have to add /Soft21/ to the path).
So I tested

<IfModule mod_rewrite.c>
RewriteEngine on
RewriteCond %{HTTP_REFERER} ^$
RewriteCond %{HTTP_REFERER} ^http://www.doom9.org.* [OR]
RewriteCond %{HTTP_REFERER} ^http://doom9.org.* [OR]
RewriteCond %{PATH_INFO} ^/usr/home/doom9/public_html/.* [OR]
RewriteRule ^/$ .*\.zip|rar|exe$ http://www.doom9.org/ [R]
</IfModule>

which gives me a 404 for each file.. so at least I know that .htaccess is allowed and mod_rewrite is installed.

adding trailing slashes to the urls above doesn't do the trick, neither does putting the full path to the software files (with trailing slash). putting the zip|rar|exe part into () brackets doesn't help either.

then I put in the following:
<IfModule mod_rewrite.c>
RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://www.doom9.org/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://doom9.org/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^/~doom9/.*$ [NC]
RewriteRule .*\.(zip|exe|rar)$ - [F]
</IfModule>
that works.. but it also works when I just copy and paste a link into an empty browser so while the downloads work again the referrer check doesn't.

after combining some of the examples given I got the following:
<IfModule mod_rewrite.c>
RewriteEngine on
RewriteCond %{HTTP_REFERER} ^$
RewriteCond %{HTTP_REFERER} ^http://www.doom9.org.* [OR]
RewriteCond %{HTTP_REFERER} ^http://doom9.org.* [OR]
RewriteCond %{HTTP_REFERER} ^http://forum.doom9.org.* [OR]
RewriteCond %{PATH_INFO} ^/~doom9/.* [OR]
RewriteRule .*\.(zip|rar|exe)$ http://www.doom9.org/leech.htm [R]
</IfModule>

which does work.
now.. if I put that on server2 (which is the one that does downloads via ftp.. guess what happens.. instead of going to ftp it gets the fila via ftp.. from www.doom9.org even though I force www.doom9.org to use the IP of server2 and you can't download any files from server2 via http. now where on earth are these files coming from?

[dRD]

We (AfterDawn.com) use the similiar method in our Apache, we have created a separate subdomain dl.afterdawn.com that is basically our download directory and we have this in our virtualhost entry in http.conf file:

RewriteEngine on
RewriteCond %{HTTP_REFERER} !^http://www.afterdawn.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://fin.afterdawn.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://*.afterdawn.com/.*$ [NC]
RewriteCond %{REMOTE_ADDR} !^64\.27\.90\.83$
RewriteRule .*$ - [F]

...works nicely, blocks all the "bad" requests to the subdomain. Only major nag with this is that "download assistants" don't work at all. And that GetRight allows users to hack the system so that it can send out anything as HTTP_REFERER which spoils the idea, but as 99% of the GetRight users are not smart enough to do anything with its special features, it really doesn't matter. By adding HTTP compression, very cryptic dynamic mirror system (checks deadlinks every 15mins) and "leechblocks" we've reduced our bandwidth from 600GB to 110GB a month (hotlinkers are not really the major problem, major problem are download assistants with mirror search and as our servers tend to be faster than, let's say, WinDVD's servers, we saw tons of people downloading WinDVD from our site instead of Intervideo's site).

...oh, the REMOTE_ADDR allows downloads from that specific IP without HTTP_REFERER -- there's a specific reason for that one to be there as well :-)

Btw. Rackshack.net -- $99/300GB/month..

[Doom9]

we're still looking into the server2 thing.. it's getting stranger and stranger every day.. just found some gknot downloads via http in the server log... but I think what happens is that for whatever reasons people get the file from another server than server2 because of the whole ftp thing (maybe because of download managers.. it takes relatively long for the ftp to kick in so in the meantime the other server would've chosen for the download which eventually lead to peaking out its connection).

There's another thing I'm fighting with atm.. redirects from domainhosts. The script given above doesn't work when you simply replace the .org with .net and paste a .net url in your browser.

any ideas?

and.. can anybody tell me what
RewriteCond %{HTTP_REFERER} !^$
does exactly? the ! negates something.. but what does the $ stand for?

[Doom9]

one more thing.. for whatever reasons.. the following does not work:
<IfModule mod_rewrite.c>
RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://www.doom9.org/.$* [NC]
RewriteCond %{HTTP_REFERER} !^http://doom9.org/.$* [NC]
RewriteCond %{HTTP_REFERER} !^http://forum.doom9.org/.$* [NC]
RewriteCond %{PATH_INFO} !^/~doom9/.$*
RewriteRule .*\.(zip|rar|exe)$ http://www.doom9.org/leech.htm [R]
</IfModule>

does not work for me, but if I remove the !'s and replace the NCs with ORs it works just fine

not even <IfModule mod_rewrite.c>
RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://www.doom9.org/.$* [NC]
RewriteCond %{HTTP_REFERER} !^http://doom9.org/.$* [NC]
RewriteCond %{HTTP_REFERER} !^http://forum.doom9.org/.$*
RewriteRule .*$ - [F]
</IfModule>
works.. as soon as I start negating it doesn't work anymore.. strange, isn't it?

[dRD]

Originally posted by Doom9

and.. can anybody tell me what
RewriteCond %{HTTP_REFERER} !^$
does exactly? the ! negates something.. but what does the $ stand for?

http://httpd.apache.org/docs/mod/mod_rewrite.html

It means "if the HTTP_REFERER is not XXXXXXXXX, then...". And $ is a backreference. So the ReWriteRule looks at all the dollars and sees if any of those matches in order to decide if it should run the rule or not. You can have multiple separate rules and conditions by having number after $, like $1 and own rule that looks for $1 conds.

[dRD]

Originally posted by Doom9
one more thing.. for whatever reasons.. the following does not work:
<IfModule mod_rewrite.c>
RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://www.doom9.org/.$* [NC]
RewriteCond %{HTTP_REFERER} !^http://doom9.org/.$* [NC]
RewriteCond %{HTTP_REFERER} !^http://forum.doom9.org/.$* [NC]
RewriteCond %{PATH_INFO} !^/~doom9/.$*
RewriteRule .*\.(zip|rar|exe)$ http://www.doom9.org/leech.htm [R]
</IfModule>

does not work for me, but if I remove the !'s and replace the NCs with ORs it works just fine

not even <IfModule mod_rewrite.c>
RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://www.doom9.org/.$* [NC]
RewriteCond %{HTTP_REFERER} !^http://doom9.org/.$* [NC]
RewriteCond %{HTTP_REFERER} !^http://forum.doom9.org/.$*
RewriteRule .*$ - [F]
</IfModule>
works.. as soon as I start negating it doesn't work anymore.. strange, isn't it?

You're turning the negates to opposite by removing the !s. And NC means "no case sensitive" and OR means "or next condition". The last line is missing tabs, it needs to have specific tabbing -- you can copypaste it from the URL I posted earlier (apache.org) to get it right. It basically just gives 403 if someone tries to access your system from outside URL or without referer.

And at least Win's Apache doesn't work when having those things in htaccess, for whatever reason, it needs to be in virtualhosts.

[Doom9]

tabbing? don't find anything about that on the apache site.. all I get by searching for tab is table a couple of times.. and if I copy and paste the examples I found (no NC stuff.. but ORs are there)

RewriteCond %{REMOTE_HOST} ^host1.* [OR]
RewriteCond %{REMOTE_HOST} ^host2.* [OR]
RewriteCond %{REMOTE_HOST} ^host3.*
RewriteRule ...some special stuff for any of these hosts...

and that has no tabbing I could see.

Furthermore.. I don't quite get why it works for me when it clearly shouldn't.. after all my script now looks like this:
<IfModule mod_rewrite.c>
RewriteEngine on
RewriteCond %{HTTP_REFERER} ^$
RewriteCond %{HTTP_REFERER} ^http://www.doom9.org/.$* [OR]
RewriteCond %{HTTP_REFERER} ^http://doom9.org/.$* [OR]
RewriteCond %{HTTP_REFERER} ^http://forum.doom9.org/.$* [OR]
RewriteRule .*\.(zip|rar|exe)$ http://www.doom9.org/leech.htm [R]
</IfModule>

and that should basically not allow any links from within my domain.. but it does the exact opposite. If I remove the [OR] in the 5th line it doesn't work at all. I also tried to putting in negations to every line, and tried this with and without the last [OR].. result.. it just doesn't work. The only way ti get it working is to put what I pasted above.
btw.. the server is running on freebsd.

[Doom9]

and another thing.. what do I do about www.doom9.net redirecting to www.doom9.org? try it: IP: 213.114.110.21, send a request for *.doom9.org to that addy (changing the host file in windows) and you'll see that while links for doom9.org don't work unless they originate from my domain, if you replace the .org with .net it does work just fine. the .htaccess file on the server is the one I pasted in my last post.

[dRD]

Originally posted by Doom9
tabbing? don't find anything about that on the apache site.. all I get by searching for tab is table a couple of times.. and if I copy and paste the examples I found (no NC stuff.. but ORs are there)

RewriteCond %{REMOTE_HOST} ^host1.* [OR]
RewriteCond %{REMOTE_HOST} ^host2.* [OR]
RewriteCond %{REMOTE_HOST} ^host3.*
RewriteRule ...some special stuff for any of these hosts...

and that has no tabbing I could see.

Furthermore.. I don't quite get why it works for me when it clearly shouldn't.. after all my script now looks like this:
<IfModule mod_rewrite.c>
RewriteEngine on
RewriteCond %{HTTP_REFERER} ^$
RewriteCond %{HTTP_REFERER} ^http://www.doom9.org/.$* [OR]
RewriteCond %{HTTP_REFERER} ^http://doom9.org/.$* [OR]
RewriteCond %{HTTP_REFERER} ^http://forum.doom9.org/.$* [OR]
RewriteRule .*\.(zip|rar|exe)$ http://www.doom9.org/leech.htm [R]
</IfModule>

and that should basically not allow any links from within my domain.. but it does the exact opposite. If I remove the [OR] in the 5th line it doesn't work at all. I also tried to putting in negations to every line, and tried this with and without the last [OR].. result.. it just doesn't work. The only way ti get it working is to put what I pasted above.
btw.. the server is running on freebsd.

Your .$* is in wrong way, should be .*$

[Freaky]

Originally posted by Doom9
and.. can anybody tell me what
RewriteCond %{HTTP_REFERER} !^$
does exactly? the ! negates something.. but what does the $ stand for? [/B]

^ matches the start of the string, $ matches the end. . matches any character (if you want to match a literal ".", use \.). * matches any number of whatever you specified previously (so .* matches anything). The . can be replaced by any character, which represents that character, except for the metacharacters {}()[].\ which need to be escaped. You can also replace it with a range of characters such as [a-z], [az], and their negations ([^a-z][^az]). Parts of the match can be saved by use of (), and referenced left to right using $1-$9. * can be replaced with {min,max} or an absolute {number}.

Are you missing the RewriteBase line? By the time .htaccess is processed, Apache is a long way into the request handler, so mod_rewrite needs to do some hackery to fix it.

Add the line:

RewriteBase /software/

or whatever the URI for the directory is. Also, don't forget you need permission to use symlinks for mod_rewrite - you'd get .htaccess errors if you don't, but it's something to remember.

[Doom9]

<IfModule mod_rewrite.c>
RewriteEngine on
RewriteBase /Soft21/
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://www.doom9.org/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://doom9.org/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://forum.doom9.org/.*$ [NC]
RewriteRule .*\.(zip|rar|exe)$ http://www.doom9.org/leech.htm [R]
</IfModule>

doesn't work either. Let's face it.. negations don't work on this server at all (and on the others they don't work either)...
let's go thru this line by line:
RewriteCond %{HTTP_REFERER} !^$ if the referer is not empty
RewriteCond %{HTTP_REFERER} !^http://www.doom9.org/.*$ [NC]
if the referer does not start with the string above (btw.. why do we put a dot after the trailing slash? wouldn't /*$ be enough to say any url that starts with...? NC means case is not important (btw.. case wasn't important with the ORs either.. I tried)
RewriteCond %{HTTP_REFERER} !^http://doom9.org/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://forum.doom9.org/.*$ [NC]
same as above for different base URLs
RewriteRule .*\.(zip|rar|exe)$ http://www.doom9.org/leech.htm [R]

redirect everything that ends with .zip, .rar and .exe to the leecher page. Why do we put a dot before the star here?


as before you can verify that it doesn't work by accessing the ip I gave you.. the script I pasted above is on the server as of now.

[c0p0n]

@Doom9
btw.. why do we put a dot after the trailing slash?

the "." stands for "everything". for example,
we have http://www.doom9.org/.$ --> http://www.doom9.org/whateverweputhere

so with the dot, we´re refering to all documents contained into doom9.org.
same also applies to RewriteRule.

let´s see a single line:

RewriteRule .*\.(zip|rar|exe)$ http://www.doom9.org/leech.htm [R]

the $ symbol needs to be placed often at the end of a line, try

RewriteRule .*\.(zip|rar|exe) http://www.doom9.org/leech.htm $ [R]

[Freaky]

Originally posted by Doom9
<IfModule mod_rewrite.c>
RewriteEngine on
RewriteBase /Soft21/
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://www.doom9.org/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://doom9.org/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://forum.doom9.org/.*$ [NC]
RewriteRule .*\.(zip|rar|exe)$ http://www.doom9.org/leech.htm [R]
</IfModule>


Aside from fairly dodgy regex there (mostly superfluous stuff), it looks ok. mod_rewrite can be a bit tempremental at times though.

How about the mostly identical:

RewriteEngine on
RewriteBase /Soft21/
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://[^/]*doom9\.org/ [NC]
RewriteRule \.(zip|rar|exe)$ http://www.doom9.org/leech.htm [L]

Which tests ok under 1.3.23:


GET /~freaky/foo.zip HTTP/1.0
Referer: http://foo.doom9.org/

HTTP/1.1 200 OK



GET /~freaky/foo.zip HTTP/1.0
Referer: http://www.foo.com/doom9.org/

HTTP/1.1 302 Found



btw.. why do we put a dot after the trailing slash? wouldn't /*$ be enough to say any url that starts with...?


No, regular expressions are not globs - /* says "match any number of forward slashes", where as ".*" says "match any number of anything". The $ matches the end of the string, so dropping that implies the .*.

http://www.aagh.net/php/regex/intro :)

[omol]

Originally posted by Doom9
and.. can anybody tell me what
RewriteCond %{HTTP_REFERER} !^$
does exactly? the ! negates something.. but what does the $ stand for?

$ is regexp special character, means end of line. ^ means begin of line. So, the above example in context means, if HTTP_REFERRER is not empty.

regards,
omol

[omol]

Originally posted by Doom9
Furthermore.. I don't quite get why it works for me when it clearly shouldn't.. after all my script now looks like this:
<IfModule mod_rewrite.c>
RewriteEngine on
RewriteCond %{HTTP_REFERER} ^$
RewriteCond %{HTTP_REFERER} ^http://www.doom9.org/.$* [OR]
RewriteCond %{HTTP_REFERER} ^http://doom9.org/.$* [OR]
RewriteCond %{HTTP_REFERER} ^http://forum.doom9.org/.$* [OR]
RewriteRule .*\.(zip|rar|exe)$ http://www.doom9.org/leech.htm [R]
</IfModule>

and that should basically not allow any links from within my domain.. but it does the exact opposite. If I remove the [OR] in the 5th line it doesn't work at all. I also tried to putting in negations to every line, and tried this with and without the last [OR].. result.. it just doesn't work. The only way ti get it working is to put what I pasted above.

The regexp construct is NOT valid. "$" means end of line, but you put a "*" after end of line? And this will turn into "end of line or more". "*" means more of the previous character. A "." means any single alphanumeric character including space.

regards,
omol

[omol]

Ah, should have read thru the thread, I guess Doom already get his answer. Many replies.....;)

regards,
omol

[yuinfo]

Originally posted by c0p0n
@Doom9
btw.. why do we put a dot after the trailing slash?

the "." stands for "everything". for example,
we have http://www.doom9.org/.$ --> http://www.doom9.org/whateverweputhere

Not true cOpOn, the "." stands for "one single character", so your string wont match unless you put the "*" after the "." meaning "repeating 0+ times". Correct example should be:

^http://www.doom9.org/.*$
and it wiil match
http://www.doom9.org/whateverweputhere

[c0p0n]

ops, of course you´re right, a little confussion before going to bed. thx for the correction :)

[Doom9]

RewriteEngine on
RewriteBase /Soft21/
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://[^/]*doom9\.org/ [NC]
RewriteRule \.(zip|rar|exe)$ http://www.doom9.org/leech.htm$ [L]
nogo
RewriteRule \.(zip|rar|exe) http://www.doom9.org/leech.htm$ [L]
nogo
RewriteEngine on
RewriteBase /Soft21/
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://*\.doom9\.org/ [NC]
RewriteRule .*\.(zip|rar|exe)$ http://www.doom9.org/leech.htm$ [R]
nogo
RewriteEngine on
RewriteBase /Soft21/
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://*\.doom9\.org/ [NC]
RewriteRule \.(zip|rar|exe)$ http://www.doom9.org/leech.htm$ [R]
nogo

here's another try:
RewriteEngine on
RewriteBase /Soft21/
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://.*\.doom9\.org/.*$ [NC]
RewriteRule ^.*\.(zip|rar|exe)$ http://www.doom9.org/leech.htm [R]
rewritecond 1 matches empty referers
rewritecond 2 matches http://*.doom9.org/* and this time as properly done as it can be
rewriterule matches *.zip *.rar and *.exe and is properly opened and closed. but of course.. it doesn't work either

but.. as with all open source software the documentation is more than crappy. http://httpd.apache.org/docs/misc/rewriteguide.html
here's one example:
RewriteRule .*\.gif$ - [F]
why don't they use the ] ^ to start a string? says to use so in the documentation.then another one: RewriteRule ^/.* - [F]
so.. this time it's properly opened but not closed.. no wonder nobody can figure it out if the author himself can't. if microsoft should focus more on security the open source community should focus more on making their things understandable. regular expressions are the best example for something that no normal human being will ever be able to truly understand and these matches here are a piece of cake compared to what you can find in some perl scripts.

so.. I'm giving up now.. I've wasted more than enough time on this. Here's one again a description of what I'm looking for:
for directory Soft21/ in my web root and all subdirectories:
check if the referer is *.doom9.net or *.doom9.org and allow access to rar, zip and exe files, if it doesn't match, redirect to www.doom9.org/leech.htm. Whoever can figure it out shall receive the Official Doom9 approved apache expert award ;)

[mat813]

Originally posted by Doom9
@xming: should that be in httpd.conf? do the first two lines of the script take care for every doc being served or does it have to be set up account specific (like
<Directory /home/doom9/public_html>
AllowOverride All
<Limit GET POST>
Order allow,deny
Allow from all
</Limit>

this is quite strange thing, I believe that it lacks an /directory to end it and that the limit thing is quite useless (limit get and post to allow it from all)

<Directory /home/doom9/public_html>
AllowOverride All
</Directory>

by the way, on my server, you already have a allowoverride all

btw, my apache is :
CFLAGS='-DSINGLE_LISTEN_UNSERIALIZED_ACCEPT -DBUFFERED_LOGS -DHARD_SERVER_LIMIT=2048' ./configure --prefix=/opt/www --enable-module=all --enable-shared=max
not all modules are enabled in the conf, but all are here.

[xming]

Originally posted by Doom9
do these rules also work for your subdirectories or do you have to put them into every directory?

@xming: should that be in httpd.conf? do the first two lines of the script take care for every doc being served or does it have to be set up account specific (like
<Directory /home/doom9/public_html>
AllowOverride All
<Limit GET POST>
Order allow,deny
Allow from all
</Limit>
and be repeated for every account that has web access?

you can guess from all this that I don't have access to the server config file so things are a bit complicated

you should put that in your httpd.conf to activate .htaccess (you if it's a old apache in access.conf)

ones you put that dir in the httpd.conf you can use .htaccess in that specific dir to do whatever you want. Oh BTW you can use wildacards eg

<Directory /home/*/public_html>
AllowOverride All
<Limit GET POST>
Order allow,deny
Allow from all
</Limit>

so every public_html in everyone's home has this feature

cheers

xming

[xming]

Originally posted by mat813


this is quite strange thing, I believe that it lacks an /directory to end it and that the limit thing is quite useless (limit get and post to allow it from all)

<Directory /home/doom9/public_html>
AllowOverride All
</Directory>

by the way, on my server, you already have a allowoverride all

btw, my apache is :
CFLAGS='-DSINGLE_LISTEN_UNSERIALIZED_ACCEPT -DBUFFERED_LOGS -DHARD_SERVER_LIMIT=2048' ./configure --prefix=/opt/www --enable-module=all --enable-shared=max
not all modules are enabled in the conf, but all are here.

It was just an example to show you can limit the post and/or get. But yes it is useless

xming :)

[xming]

Oh jee, I can't make it in one post. doom9 you can put the <Limit></Limit> in your .htaccess

[Freaky]

Originally posted by Doom9
RewriteEngine on
RewriteBase /Soft21/
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://[^/]*doom9\.org/ [NC]
RewriteRule \.(zip|rar|exe)$ http://www.doom9.org/leech.htm$ [L]
nogo


Um, where did the $ at the end of the rewriterule come from?


RewriteRule \.(zip|rar|exe) http://www.doom9.org/leech.htm$ [L]
nogo


Moving characters around at random won't help :)


RewriteCond %{HTTP_REFERER} !^http://*\.doom9\.org/ [NC]


This opens a hole - it allows refering URL's to contain "doom9.org/" anywhere in the address and leech away.


RewriteRule .*\.(zip|rar|exe)$ http://www.doom9.org/leech.htm$ [R]


The .* at the start is superfluous. Without a ^ an anchor it to the start of the line, the regex doesn't care what it starts with.


here's another try:
RewriteEngine on
RewriteBase /Soft21/
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://.*\.doom9\.org/.*$ [NC]


The .*$ is superfluous. Without a $ to anchor it to the end of the string, the regex doesn't care what's there.


RewriteRule ^.*\.(zip|rar|exe)$ http://www.doom9.org/leech.htm [R]
rewritecond 1 matches empty referers
rewritecond 2 matches http://*.doom9.org/* and this time as properly done as it can be


This allows referers such as http://www.foo.bar.com/wibble/doom9.org/bla. That's why I used [^/].


rewriterule matches *.zip *.rar and *.exe and is properly opened and closed. but of course.. it doesn't work either


The regex doesn't need to be opened or closed.


but.. as with all open source software the documentation is more than crappy.


Looks quite thorough to me.


http://httpd.apache.org/docs/misc/rewriteguide.html
here's one example:
RewriteRule .*\.gif$ - [F]
why don't they use the ] ^ to start a string?


The string doesn't need to be started, and the .* is superfluous. I guess he was just being explicit when he wrote that.


says to use so in the documentation.


Where does it say every regex needs to be anchored at either end? They're delimited by whitespace that is not contained within a character class (such as [ ]), not ^ and $.


then another one: RewriteRule ^/.* - [F]
so.. this time it's properly opened but not closed..


It doesn't need to be closed, ^/ says "match any URL starting with /" and the .* just makes it clearer why there's no end of string anchor.


regular expressions are the best example for something that no normal human being will ever be able to truly understand and these matches here are a piece of cake compared to what you can find in some perl scripts.


Regex is a bit like maths - it may look complex if you've never done it before, but it really isn't (well, the basic operations aren't). What is bad is guessing what things mean without at least scanning over the manpages to find out what the basics are.

I suspect:

[list=1]
You're using a Windows text editor and embeding CRLF at the end of lines instead of just LF.
mod_rewrite isn't enabled, your <IfModule> line is hiding it, and it's silently doing nothing.
Options FollowSymLinks is not set in httpd.conf, so you don't have permission to use mod_rewrite.
AllowOverride All or AllowOverride FileInfo is not set in your httpd.conf for the directory and it is failing.
The .htaccess file is unreadable by Apache because of permissions, or maybe the name has been changed in httpd.conf (i.e. under Win32 it's probably _htaccess because of explorer brokenness).
[/list=1]


so.. I'm giving up now.. I've wasted more than enough time on this. Here's one again a description of what I'm looking for:
for directory Soft21/ in my web root and all subdirectories:
check if the referer is *.doom9.net or *.doom9.org and allow access to rar, zip and exe files, if it doesn't match, redirect to www.doom9.org/leech.htm. Whoever can figure it out shall receive the Official Doom9 approved apache expert award ;)

Right:


Connected to localhost.
Escape character is '^]'.
GET /~freaky/foo.zip HTTP/1.0
Referer: foobar

HTTP/1.1 302 Found
Date: Tue, 05 Feb 2002 16:16:27 GMT
Server: Apache/1.3.23 (Unix) DAV/1.0.2 PHP/4.1.1
Location: http://www.doom9.org/leech.htm

GET /~freaky/foo.zip HTTP/1.0
Referer: http://forums.doom9.net/foo.php?bar&wibble

HTTP/1.1 200 OK
Date: Tue, 05 Feb 2002 16:16:54 GMT
Server: Apache/1.3.23 (Unix) DAV/1.0.2 PHP/4.1.1
Last-Modified: Tue, 05 Feb 2002 01:41:22 GMT


My Apache config for public_html directories includes "AllowOverride FileInfo" and "Options FollowSymLinks".

The entire .htaccess file is:


RewriteEngine on
RewriteBase /~freaky/
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://[^/]*doom9\.(org|net)/ [NC]
RewriteRule \.(zip|rar|exe)$ http://www.doom9.org/leech.htm [L]


Line endings are Unix LF, each field is seperated by exactly one space, there is no surrounding <IfModule> directive, and this is on a FreeBSD 4.5-PRERELEASE box running Apache, mod_dav and mod_php from ports.

In fact, if you do:


curl http://freak.aagh.net/code/doom9.htaccess >.htaccess


In the directory, it should Just Work, assuming the httpd.conf prerequisites are fulfilled.

If not, shove my SSH public key in ~/.ssh/authorized_keys and I'll log in and try to find what's up ;)

[Doom9]

!^http://[^/]*doom9\.(org|net)/ [NC]
none of / must be between the :// and doom9 if I get this right. but how about www.webdoom9.org.. that would work, wouldn't it? don't you have to force the dot before doom9 as well? so like

!^http://[^/]*\.doom9\.(org|net)/ [NC]

also.. the server is freebsd 4.4, but I write these files with winodws notepad. all the httpd.conf options should be set since it does work with the [OR]example I posted and depending on where I mess around I only get 404s. So, mod_rewrite is also definitely there.. the problem is the script no the server conf. How to convert a doc from the dos world to the unix world? I know dos2unix and the other way round.. but these are *nix tools

and why do you use [L] instead of [R]?

and.. I have another note on the doc.. why make delimiters, dots and stars if you can leave them away? either force it or leave it would make more sense. I've taken math courses for 16 years now but regular expressions.. they simply don't make any sense unless you have a gordian knot in your brain.. I've learned perl and I managed basic regexps but in the O'reilly book there's an expression to check an url for validity.. and I really doubt that many people fully understand what each char does there.

[Freaky]

Originally posted by Doom9
none of / must be between the :// and doom9 if I get this right. but how about www.webdoom9.org.. that would work, wouldn't it? don't you have to force the dot before doom9 as well? so like

!^http://[^/]*\.doom9\.(org|net)/ [NC]


Well, that breaks just looking for http://doom9.org/, but at least you're thinking in the right direction :P

!^http://([^/]*\.)?doom9\.(org|net)/ [NC]

Matches (optionally [any number of characters that are not /] full stop) doom9. (org or net)/


also.. the server is freebsd 4.4, but I write these files with winodws notepad.


Blegh, use a real text editor - personally I use http://www.vim.org/, but even http://www.textpad.com/ beats crap out of notepad. Hell, cat >foo.txt beats crap out of notepad ;)


How to convert a doc from the dos world to the unix world? I know dos2unix and the other way round.. but these are *nix tools


TextPad and Vim include these options. Why aren't you editing them in-place on the server with a Unix editor? :)


and why do you use [L] instead of [R]?


The manual states:

Prefix Substitution with http://thishost[:thisport]/ (which makes the new URL a URI) to force a external redirection.


But since you're already giving a full URI, it's superfluous. [L] just defines it as being the last rule so any future rewrite rules are not applied.


and.. I have another note on the doc.. why make delimiters, dots and stars if you can leave them away? either force it or leave it would make more sense.


*shrug*, doesn't make more or less sense, they both mean the same thing. They're not delimiters - they're part of the regexp, the spaces are delimiters.


I've taken math courses for 16 years now but regular expressions.. they simply don't make any sense unless you have a gordian knot in your brain..


Well, my maths sucks ass, but regex is rather transparent to me. I guess I'm just weird :)


I've learned perl and I managed basic regexps but in the O'reilly book there's an expression to check an url for validity.. and I really doubt that many people fully understand what each char does there.


Well, if you can think of a better way of doing it that doesn't involve writing full parsers or manually writing string scanners, go ahead :)

[Marwin]

Hi

Webmaster of www.divxmovies.com here. Haven't really analyzed the example you are trying to use, but figured I could share how we do our referer checking. We've simply created an .htaccess file in the download directory which contains the following lines:

RewriteEngine on

RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://.*divxmovies.com/.*$ [NC]
RewriteRule .*[Ee][Xx][Ee]$ http://www.divxmovies.com/software/ [R]

RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://.*divxmovies.com/.*$ [NC]
RewriteRule .*[Rr][Aa][Rr]$ http://www.divxmovies.com/software/ [R]

RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://.*divxmovies.com/.*$ [NC]
RewriteRule .*[Zz][Ii][Pp]$ http://www.divxmovies.com/software/ [R]

The only problem as I see it would be if you don't have the RewriteEngine enabled in your httpd.conf, but it was there by default on our webserver.

[omol]

Originally posted by Freaky
TextPad and Vim include these options. Why aren't you editing them in-place on the server with a Unix editor?

I guess Doom already mentioned before, he only has ftp access to the www box.

ps. hope there won't be another vi vs emacs thread here....;)

regards,
omol

[Doom9]

@freaky: finally the linux freak I was looking for... but, you may well be right with your script but it doesn't work on my box.
I created the file in textpad, I removed all the trailing empty spaces you get when copying from the browser (also tried to keep them).. with negations it doesn't work, if I remove the negations and put an [OR] behind the *.doom9.org check it does work.

I saved the following .htaccess file using UNIX ansi format in TextPad 4.5.0
RewriteEngine on
RewriteBase /~doom9/public_html/Soft21/
RewriteCond %{HTTP_REFERER} ^$
RewriteCond %{HTTP_REFERER} ^http://([^/]*\.)?doom9\.(org|net)/ [OR]
RewriteRule \.(zip|rar|exe)$ http://www.doom9.org/leech.htm [L]

put a ! before both RewriteCond's and replace the [OR] with an NC and it no longer works. So.. we still haven't progressed very far.. our regexp is not close to perfection.. but the negation thing still doesn't work. Also.. putting [L] instead of [R] loads the right html file.. but the download link is still in the browser and it screws up the css file that the html page is supposed to load (apparently it doesn't load any).

[Doom9]

also.. the net condition still doesn't work.. I don't doubt it would work if .net was a real domain, but it's just a web forward to www.doom9.org courtesy by namezero.. maybe you can analyze the output from the redirect and come up with something.

[Doom9]

I also have a linux server (sorry, dunno what os they're using) that I use for testing purposes (server5.doom9.org) where the negations don't work either.

[Rasqual]

Originally posted by Doom9
RewriteEngine on
RewriteBase /~doom9/public_html/Soft21/
RewriteCond %{HTTP_REFERER} ^$
RewriteCond %{HTTP_REFERER} ^http://([^/]*\.)?doom9\.(org|net)/ [OR]
RewriteRule \.(zip|rar|exe)$ http://www.doom9.org/leech.htm [L]

put a ! before both RewriteCond's and replace the [OR] with an NC and it no longer works. So.. we still haven't progressed very far.. our regexp is not close to perfection.. but the negation thing still doesn't work. Also.. putting [L] instead of [R] loads the right html file.. but the download link is still in the browser and it screws up the css file that the html page is supposed to load (apparently it doesn't load any).

Sounds good... but let's be reasonable:
If the referer is from doom9 => do NOT rewrite
If the referer isnt fm doom9 => *do* rewrite

Conditions for NOT rewriting are:
Referer is "" OR Referer is from doom9
That's why...
Conditions for rewriting are:
NOT (Referer is "") AND NOT (Referer is from doom9)
(De Morgan's law)

Please try with a computer syntax.

[Doom9]

i found the following in the mod_regexp documentation:
Notice: When using the NOT character to negate a pattern you cannot have grouped wildcard parts in the pattern. This is impossible because when the pattern does NOT match, there are no contents for the groups. In consequence, if negated patterns are used, you cannot use $N in the substitution string!that could have something to do with our problems negating, couldn't it (we know that if we don't negate we get a match)... the () brackets are for grouping and we have a wildcard in the group

[Rasqual]

Was there a NOT operator? Huh, I didn't know ^_^;

What I meant was ! but I put NOT to make it more explicit.

[mat813]

I may be wrong, but the difference between NOT and ! is unclear to me :)

[Rasqual]

RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://([^/]*\.)?doom9\.(org|net)/.*$ [NC]
RewriteRule .* http://www.doom9.org/leech.htm [R]


Instructions:
. Add the !
. Add .* after the "base" address http://???.doom9.org/ (match any page under this place.
. Rewrite rule is simplified (do you really want to forbid only archives? Are there other file types in the folder?)

Optional:
(:80)? Optional port

(Plz Deactivate 'Automatically parse URLs' for this post)

[Doom9]

finally it's working:
RewriteEngine on
RewriteBase /~doom9/public_html/Soft21/
RewriteCond %{HTTP_REFERER} !^http://([^/]*\.)?doom9\.(org|net)/
RewriteRule \.(zip|rar|exe)$ http://www.doom9.org/leech.htm [R]

now.. I need something else.. atm we have if it's not *.doom9.org we go to leech.htm, now I need the same plus if it's *.doom9.org I need to replace the server part with another url.

Here's an example:
If the link is http://www.doom9.org/Soft21/Audio/BeSweet1.11.zip and if the referrer matches the condition above then it should redirect to say http://anothersite.tld/somepath/Soft21/Audio/BeSweet1.11.zip

hope that's clear. atm we use the following in the .htaccess file:

RedirectPermanent /Soft21 http://anothersite.tld/somepath/

so.. how to combine that with the referer check?

[nek]

Originally posted by Doom9
Here's an example:
If the link is http://www.doom9.org/Soft21/Audio/BeSweet1.11.zip and if the referrer matches the condition above then it should redirect to say http://anothersite.tld/somepath/Soft21/Audio/BeSweet1.11.zip

hope that's clear. atm we use the following in the .htaccess file:

RedirectPermanent /Soft21 http://anothersite.tld/somepath/

so.. how to combine that with the referer check? [/B]

i guess you should try :

RewriteRule ^/Soft21/(.*)$ http://anothersite.tld/somepath/$1 [R]

(.*) matches anything after /Soft21/ $1 appends what was matched

regards

[Doom9]

so I should use something like
RewriteEngine on
RewriteBase /~doom9/public_html/Soft21/
RewriteCond %{HTTP_REFERER} !^http://([^/]*\.)?doom9\.(org|net)/
RewriteRule \.(zip|rar|exe)$ http://www.doom9.org/leech.htm [R]

RewriteCond %{HTTP_REFERER} ^http://([^/]*\.)?doom9\.(org|net)/
RewriteRule ^/Soft21/(.*)$ http://anothersite.tld/somepath/$1 [R]

is that right?

[nek]

you don't even need the

RewriteCond %{HTTP_REFERER} ^http://([^/]*\.)?doom9\.(org|net)/

line,

it will implicitly be an "else" since the first rewrite rule won't be matched

hope this works

[Rasqual]

Here's a masqueraded URL:
&nbsp;http://www%2emultimania%2ecom%2fskra%2f.doom9.org/stealer.html (copy and paste in your browser)
which translates to:
&nbsp;http://www.multimania.com/skra/.doom9.org/stealer.html
The latter does pass, not the former.

I suggest putting directly the authorized vhosts:
RewriteCond %{HTTP_REFERER} !^http://((www|srvX|whatever)\.)?doom9\.(org|net)/

---
{Edit Note}
The .htaccess protected server is server3,
phpinfo is here to check out the referer (lazy to write a script ^_^; )

[tCS [hF]]

hey folks,

since allmost anything has been said already, my final tip is to enable debugging while writing regular expressions with mod_rewrite. to do this, include the following statement in your conf:

RewriteEngine on
RewriteLog "/www/logs/rewrite.log"
RewriteLogLevel 0

put the rewriteloglevel to something reasonable, say 3, then you'll see how your request is being rewritten. this saves a _lot_ of time while searching for possible bugs.

a look at some regular expression book say this one (http://www.amazon.com/exec/obidos/ASIN/1565922573/qid=101) might help, or at least the unix manpages regex(3) and regex(7).

HTH
tCS

[Doom9]

@rasqual: I don't get your point.. both these urls don't pass your referer check test with either regexp.. and the one we're using now:
!^http://([^/]*\.)?doom9\.(org|net)/

says either doom9.org/net, or *.doom9.org/net where the * must not include a / so it can't be a subdir on a server

[Doom9]

@nek: I'm afraid that won't do it. Here's what I have now:

RewriteEngine on
RewriteBase /~doom9/public_html/Soft21/
RewriteCond %{HTTP_REFERER} !^http://([^/]*\.)?doom9\.(org|net)/
RewriteRule \.(zip|rar|exe)$ http://www.doom9.org/leech.htm [R]

RewriteCond %{HTTP_REFERER} ^http://([^/]*\.)?doom9\.(org|net)/
RewriteRule ^/Soft21/(.*)$ ftp://ftp.doom9.org/$1 [R]

while it blocks unauthorized access, the download still comes from server2.doom9.org and not from ftp.doom9.org as it should. I tried leaving the rewritecond before the 2nd rewriterule away but that didn't help either.

So I'm back at the .htaccess we currently have:

RedirectPermanent /Soft21 ftp://ftp.doom9.org/

So.. how can I combine these two? I want the ftp redirect to only apply if access is allowed, but then it must be rewritten.

[Rasqual]

Originally posted by Doom9
@rasqual: I don't get your point.. both these urls don't pass your referer check test with either regexp.. and the one we're using now:
!^http://([^/]*\.)?doom9\.(org|net)/

says either doom9.org/net, or *.doom9.org/net where the * must not include a / so it can't be a subdir on a server

That's why the / has been masqueraded:
In http://www%2emultimania%2ecom%2fskra%2f.doom9.org/stealer.html,
the part www%2emultimania%2ecom%2fskra%2f. does match ([^/]*\.)? .
As you said, in the *.doom9.org pattern, * cannot have "/" that's why they're hidden, changed into %2f!

Please note that this method will _not_ work on Netscape 4.75 but has successfully passed in IE5 and IE6.

[Maz]

I'm no expert in mod_rewrite, but looking at their examples, it seems you should write :
RewriteRule ^/Soft21(.*)$ ftp://ftp.doom9.org$1 [R]

if this doesn't work, you should also try
RewriteRule ^/Soft21(.*) ftp://ftp.doom9.org$1 [R]

RewriteRule ^/Soft21(.*)$ ftp://ftp.doom9.org:21$1 [R]

(exactly ftp://ftp.doom9.org:21/pub/doom9$1, if I get it right)

On another hand, you should remove the trailing slash if there is one from the RedirectPermanent.

Hopes this helps

Maz

[Doom9]

maz: thanks for the suggestion but that can't be it either. IF it were a problem of pathnames the downloads would not work.. you'd get an url like ftp://ftp.doom9.org//... but.. there's no redirect taking place in the first place

[Lunatic]

:confused: I don't understand, why you think, other pages will link to your files at your HTTP-Server which you try to prevent, when they could link to them at your FTP-Server where they are not protected ... :confused:

[mat813]

well, they are not protected, but :
1)
try to browse ftp.doom9.org:/pub/doom9

2)
I host/admin the ftp server, and I don't really care for bandwidth

[Lunatic]

There is no need for browsing the directory at the ftp server. The Apache webserver would tell the browser (or an evil webmaster who's stealing links) the appropriate path/filename when doing the redirect.

I think it makes no sense to do a referer check, when you keep the ability to get the files directly by ftp. The referer check consumes cpu time while not doing anything useful, since only links by doom9 would be checked (because links from others would point to the ftp when they've once realized the links won't work over http).

[jr]

Pretty much only way to make sure others don't steal links to FTP server is use separate account and change password often. Then you just change ftp://ftp.doom9.org/file.zip to ftp://user:pass@ftp.doom9.org/file.zip.

Alternate option would be moving files around on FTP server. Eg. ftp://ftp.doom9.org/alsjdsk/file.zip and changing it to ftp://ftp.doom9.org/kksdlks/file.zip couple days later.

Either one will mean that any proxies between user and FTP server would be useless which means more traffic on FTP server.

[Lunatic]

I don't think renaming is a good idea ...

It's not too difficult to write a little script that gets the current URLs from doom9.org once a day. If the files are accessible somewhere without referer check (ftp), someone will find a way to link to them.

I think the best idea is to serve the files via http.

[Rasqual]

@Lunatic:
Your remark is correct, but what is actually more problematic is the quantity od used bw, not the CPU usage. As for the http redirect to the ftp, I also think the same way: it can easily be circumvened.

What would be better is to change the address dynamically every day (that would imply tight association between Webserver and FTP).
What would be even more interesting is some kind of Kleenex&reg; login/pass, that could be used only once (and then made invalid) and which have a limited lifespan. Such systems have been seen on irc if I remember well, and I liked the idea. Another problem would then be raised: how to generate a login for each download, how to manage this from the FTP server's point of view (might be specific), etc.
I must admit I don't know how all this would be done.

Any suggestions?

[mat813]

well, opie ftp access would be possible, just a bit painful to put up, I've already though of such things I would not be to difficult with pam, but someone else would have to look into it because days only have 24 hours in here and I don't have time for it :)

[jr]

Generating temporary ftp accounts with CGI isn't that difficult if you have necessary rights on server. However I don't think Doom9 has enough rights to create FTP accounts so that approach won't work.

How about this:

1. User clicks link on webpage and is sent to http://server9.doom9.org/files/HugeDownload.zip

2. CGI on server9 checks referer and other headers

3. CGI creates directory under /files which is in 0111 mode to prohibit directory listing. [ eg. ftp://ftp.doom9.org/files/kakakakfjdk/ ]

4. CGI creates hardlink of HugeDownload.zip under /files/kakakakfjdk/ directory

5. CGI returns either 302 pointing to ftp://ftp.doom9.org/files/kakakakfjdk/HugeDownload.zip or generates webpage with ftp link. I'd prefer latter one since I often surf web with one PC and download files to my unix shell that has faster connections and then transfer files to home while sleeping..

6. Cleanup process removes hardlinks and folders every now and then. Perhaps every hour or two.


There's still same problem with someone creating script that generates download requests for files, goes thru CGI to get final FTP url and passes that URL to user as their own download site.

Even if someone would create script that generates one-time FTP accounts it's possible to write script that fetches list of those OTP FTP accounts and passes them forward.

Of course it's a easy to ban IP such script would use. Next thing malicious webmaster might do is using some free proxy list to come thru different IP each time..

Only way to make sure no-one steals your bandwidth is using HTTP downloads or making FTP link stealing hard enough so lame webmasters don't have time nor skills to defeat it. There's always someone that will break thru it but that's small minority, right?

[mat813]

Originally posted by jr
1. User clicks link on webpage and is sent to http://server9.doom9.org/files/HugeDownload.zip

2. CGI on server9 checks referer and other headers

3. CGI creates directory under /files which is in 0111 mode to prohibit directory listing. [ eg. ftp://ftp.doom9.org/files/kakakakfjdk/ ]

4. CGI creates hardlink of HugeDownload.zip under /files/kakakakfjdk/ directory

5. CGI returns either 302 pointing to ftp://ftp.doom9.org/files/kakakakfjdk/HugeDownload.zip or generates webpage with ftp link. I'd prefer latter one since I often surf web with one PC and download files to my unix shell that has faster connections and then transfer files to home while sleeping..

6. Cleanup process removes hardlinks and folders every now and then. Perhaps every hour or two.

looks like very long for me, here is a simpler approach :
1. user clicks on http://server9.doom9.org/files/HugeDownload.zip

2. CGI creates a one time account on the ftp and redirects the browser

3. when the client has logged onto the ftp, the account is automatically disabled

end of it :)

[Rasqual]

Originally posted by mat813


looks like very long for me, here is a simpler approach :
1. user clicks on http://server9.doom9.org/files/HugeDownload.zip

2. CGI creates a one time account on the ftp and redirects the browser

3. when the client has logged onto the ftp, the account is automatically disabled

end of it :)

Yup, and clean temp account after a given amount time has passed. Any stealer may write a tool to generate fake http requests just to get any number of ftp pass.

Now, what if a webmaster uses a script or program on his own server that would connect thru http, send a fake referer field (matching a correct doom9 of course) and redirect the user to the address Apache sent back? php with sockets might do, I think.

[cyberrobo]

hu!

2 theoretical solutions for your problem

a) some php scripts which set/read cookies on the client machine (data gets written on the client machine when visiting the news/download/whatever section, which gets compared when trying to download stuff from the server (also via another script)). afaik the download scripts included in phpnuke (or at least a hack for it) provide these functionality

b) get a ftpd which is capable of authenticating users via a (my)sql-database (like proftpd). this is needed because creating "real" system accounts on the fly is quite painful (slow) and somewhat insecure. the accounts should be created on the fly when the user retrieves the page with the download links, the password protection should be invisible to the (l)user (embedded in the download link like ftp://foo:bar@ftp.doom9.org/ficken.zip). these "temporary" accounts should get invalidated after a short time-period (30 mins?).

The first solution works well on some major sites which are similar to yours (providing free (semi-illegal) information for a "small" computer-related scene), the latter is only needed when you have to provide the information/content via ftp.

if you have further questions you can contact me at research (at) amd (dot) co (dot) at, because i don't think i've got the time to follow this thread/forum

mfg michael