Direct Linking / users hammering problems again

[Koepi]

Hi people,

I'd really appreciate if you could help me out with those jerk sites (and maybe the one or the other users finds himself posted here) direct linking to my binaries.
The problem is, their (dumb!) users put that link into their download managers and let them hammer >10 times/sec onto my server no matter if they get a 403-access forbidden or not.

I'm really tired of banning whole IP ranges like i did now and add every 30 minutes a new range. If you like, dDoS those wankers, they really deserve it! Let them feel what they do to me! (And in the long end: to you, becaus eI'm again thinking about taking down my site).

Here the list-of-jerks-of-Xmas 2002:

218.58.83.244 - - [25/Dec/2002:16:18:00 +0100] "GET /~koepi/XviD-04102002-1.exe HTTP/1.1" 403 4233 "http://dvdrip.myserver.org:8199/cgi-bin/topic.cgi?forum=19&topic=232&start=0#bottom" "Mozilla/4.0 (compatible; MSIE 5.00; Windows 98)"
(extreme hammering, and the referer site produces some "hits")

ip031-310.dialup.edisontel.com - - [25/Dec/2002:16:27:48 +0100] "GET /cgi-bin/fetch?file=XviD-18082002-1.exe HTTP/1.1" 302 637 "http://www.tuttogratis.it/cgi/result.cgi" "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Hotbar 4.1.8.0)"
(that site really stinks. take it down! Produces ~10 hammering jerks per hour)

d3o840.telia.com - - [25/Dec/2002:16:33:26 +0100] "GET /~koepi/XviD-04092002-1.exe HTTP/1.0" 403 4233 "http://divxstation.com/softwareId.asp?sId=125&svId=141" "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98)"
(same goes for these: nuke these bastards!)

cm203-168-193-2.hkcable.com.hk - - [25/Dec/2002:17:27:33 +0100] "GET /~koepi/XviD_Options_Explained-pdf.zip HTTP/1.1" 403 4233 "-" "Mozilla/4.0 (compatible; MSIE 5.00; Windows 98)"
(no referer, but nuke that user - or notify him that he's a *peep*)

mail.mareco.hu - - [25/Dec/2002:17:27:38 +0100] "GET /%7Ekoepi/SMBDownLoader.exe HTTP/1.0" 403 4233 "-" "WebDownloader for X 2.4beta2"
mail.mareco.hu - - [25/Dec/2002:17:27:39 +0100] "GET /%7Ekoepi/xvid-04102002-src.tar.bz2 HTTP/1.0" 403 4233 "-" "WebDownloader for X 2.4beta2"
[...] (extreme leecher/hammering wanker - "distirbuted pings" *whistle* appreciated against him)

Ok, this list may grow. So please take a look some more often here.
I don't ask anyone to do crimial stuff, just show them that this behaviour is not tolerable - and now that i wanted to have some enjoyable spare time i have even more work because of these "sons of bitches".

So please, gimme a Xmas present :)

Thanks for all your help,
best regards
Koepi

[omol]

Originally posted by Koepi
Hi people,

218.58.83.244 - - [25/Dec/2002:16:18:00 +0100] "GET /~koepi/XviD-04102002-1.exe HTTP/1.1" 403 4233 "http://dvdrip.myserver.org:8199/cgi-bin/topic.cgi?forum=19&topic=232&start=0#bottom" "Mozilla/4.0 (compatible; MSIE 5.00; Windows 98)"
(extreme hammering, and the referer site produces some "hits")



cm203-168-193-2.hkcable.com.hk - - [25/Dec/2002:17:27:33 +0100] "GET /~koepi/XviD_Options_Explained-pdf.zip HTTP/1.1" 403 4233 "-" "Mozilla/4.0 (compatible; MSIE 5.00; Windows 98)"
(no referer, but nuke that user - or notify him that he's a *peep*)



These 2 are proxy/cache. I don't think you can do much about it. The 1st one is located in mainland China. Heavily guarded as you can expect. For the 2nd one, it's one of the local major ISPs here in Hong Kong. A very pricey Netscape proxy.

regards,
omol

[Koepi]

Thanks for the info omol,

but that clearly indicates a misconfiguration of those proxies... a 403 shouldn't start a hammering thread on a proxy (believe me, I setup proxies myself and it's really easy to set them up correctly).

So they should be notified at least that they run a misconfigured service there (maybe from hundreds of chinese doom9-users at the same time? ;) )

Regards
Koepi

[omol]

Originally posted by Koepi
but that clearly indicates a msconfiguration of those proxies... a 403 shouldn't start a hammering thread on a proxy (believe me, I setup proxies myself and it's really easy to set them up correctly).

Yup, it's not uncommon here in Greater China area, i.e. Hong Kong, Mainland China, Macau and Taiwan. The admins just simply don't care (state property), or too under-paid to be knowledgeble to handle that. If you know that Hong Kong is the major launch pad for relaying DDoS attacks, you wouldn't be surprised....;)

regards,
omol

[Koepi]

218.48.60.47 - - [25/Dec/2002:18:15:48 +0100] "GET /cgi-bin/fetch?file=XviD-08082002-1.exe HTTP/1.1" 302 637 "http://divx2002.wo.to/zboard/view.php?id=util&page=1&sn1=&divpage=1&sn=off&ss=on&sc=on&select_arrange=headnum&desc=asc&no=11" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"

That site still hasn't corrected those links. Maybe they're a worthy target as well, as that seems to be a "newbie" board and the users which use those links really hammer badly on my poor router, causing much traffic, work and thus are criminally stealing _MONEY_ in the end (less the users, but that site is responsible in this case).

pa250.nowysacz.cvx.ppp.tpnet.pl - - [25/Dec/2002:18:28:28 +0100] "GET /~koepi/XviD-04102002-1.exe HTTP/1.1" 403 4571 "-" "Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows ME) Opera 7.0 [en]"

(yet some more polish people misbehaving - hammering even if they get a 403. Seems quite usual there *grr* The whole subnet (i.e. pa33.nowysacz... does that too) is full of such freaks.)

EDIT:
pa100.krobia.sdi.tpnet.pl - - [25/Dec/2002:18:58:43 +0100] "GET /~koepi/XviD-Dec-231002.exe HTTP/1.1" 403 4580 "http://savx.w.interia.pl/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"

(see? )

hse-toronto-ppp185854.sympatico.ca - - [25/Dec/2002:18:28:48 +0100] "GET /cgi-bin/fetch?file=XviD-04102002-1.exe HTTP/1.1" 302 637 "http://www.wxp21.com/zboard/zboard.php?id=dataup&page=2&sn1=&divpage=1&sn=off&ss=on&sc=on&select_arrange=hea
dnum&desc=asc&no=7935" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"

What to say? Just more idiots.

213-97-142-4.uc.nombres.ttd.es - - [25/Dec/2002:19:08:37 +0100] "GET /~koepi/XviD-Dec-231002.exe HTTP/1.1" 403 4580 "http://www.spanishare.com/modules.php?op=modload&name=XForum&file=viewthread&tid=68286" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705)"

*grmblfx*

[NuclearFusi0n]

perhaps new builds should be circulated by bittorent? I use it all the time, it rocks. :)

http://bitconjurer.org/BitTorrent/

[Koepi]

he, no way ;)

But, again some direct linkers - these guys should really pay for the damage they do:

l85nns.lek.ru - - [25/Dec/2002:21:37:04 +0100] "GET /~koepi/XviD-24122002-1.exeHTTP/1.1" 403 4571 "http://forum.ixbt.com/0029/010722-2.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MyIE2)"

I really don't want to take down my xvid stuff, but those jerks produce so much work and traffic (which costs real money) that I'll try another week now and if it doesn't change, we have one binary distributor less.

It's a shame that even on Christmas nobody cares what they do :(

Have a nice evening,
Koepi

[StorMWinD]

sorry to hear that...check your inbox dude

:(

[ErMaC]

Sadly I think most people are just ignorant of the damage they cause. I doubt they go into this thinking "haha I'm gonna steal this sucker's bandwidth!" - they're just uneducated about the realities of the internet. That doesn't justify what they do, and doesn't mean you shouldn't ban them, but I don't think it's that they don't care or are doing it out of malice, just that they're doing it out of ignorance.

That said, you could always just start posting your binaries here at Doom9 as attachments (builds are usually less than 500K) and it seems like Doom9 has pretty good leech-blockers in place. Of course, if his leech blockers will do no better than yours then I don't recommend making the forums a target of the spamming, but I suspect the forum here is more "low-profile" than your own website, since the forum stuff doesn't show up on Google and the like.

[Koepi]

yg.gb.com.cn - - [26/Dec/2002:05:25:41 +0100] "GET /~koepi/XviD-24122002-1.exe HTTP/1.0" 403 4233 "http://popgo.net/bbs/showthread.php?s=&threadid=84506" "Mozilla/4.0 (compatible; MSIE 5.00; Windows 98)"

kill these bastards! they do it again and again. Pay the owner of that site a visit and be cruel to him (you know, try to give him some accupuncture [but only if you have no education in that matter, he shall suffer]).

EDIT:
3.47.30.61.isp.tfn.net.tw - - [26/Dec/2002:11:08:27 +0100] "GET /cgi-bin/fetch?file=XviD-09122002-1.exe HTTP/1.0" 302 559 "http://vbb.learnhome.com/bbs/showthread.php?s=&threadid=31246&highlight=%B8%D1%BDX" "Mozilla/5.0 (Windows XP; U) Opera 6.03 [zh-tw]"

That's an evil board as well...

Thanks for your help and your kind words, that helps a little for compensating for the plenty of work these *$§%/$§"/* produce...

Best regards
Koepi

[Koepi]

And yet another &%(/%!

pd9e4a068.dip.t-dialin.net - - [26/Dec/2002:15:01:31 +0100] "GET /cgi-bin/fetch?file=XviD-04102002-1.exe HTTP/1.1" 302 637 "http://members.chello.nl/t.kaspers/index/iframewarez.htm" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"

Should report that guy to the anti piracy group ;) and: since when is xvid warez?

Regards
Koepi

[wasbeer666]

Originally posted by ErMaC
Sadly I think most people are just ignorant of the damage they cause. I doubt they go into this thinking "haha I'm gonna steal this sucker's bandwidth!" - they're just uneducated about the realities of the internet. That doesn't justify what they do, and doesn't mean you shouldn't ban them, but I don't think it's that they don't care or are doing it out of malice, just that they're doing it out of ignorance.

That said, you could always just start posting your binaries here at Doom9 as attachments (builds are usually less than 500K) and it seems like Doom9 has pretty good leech-blockers in place. Of course, if his leech blockers will do no better than yours then I don't recommend making the forums a target of the spamming, but I suspect the forum here is more "low-profile" than your own website, since the forum stuff doesn't show up on Google and the like.

Dear Ermac has a point here. Most of the people aren't aware of the havoc they cause, this goes parallel with the total ignorance they and their potential distributer's posses. It's just gimme, gimme, gimme morality without realising the consequences. Unfortunately, this has it's drawback on the ones who are trying to be really productive for the audience.

However, I don't think this thread would gain you much. First of all it's very time-consuming to update it[time which you can obviously use better;) ] and it's, more or less, only the tip of the iceberg. At the end, it will indefinately result in putting down your site. Which of course nobody wants. In addition to Ermac I would suggest a more preventive solution like putting the direct link into your sig or, as stated above, attach it [or a PM for the link, but that would get you another shitload of homework:D ]. If possible, seperate the binaries and site with respect to server. In addition, try to find someone who would like to serve them for you with a server dedicated mostly for your binazries. After all, as xvid is getting more popular by time and thus with increasing demand it would be wise, at least IMHO, to outspent [if this is correct english?:)] some of your additional work which comes with the programming. Don't forget, your gift to the community is programming which you'll probably do with alot off affection/dedication. Don't let this little thing get to you and spoil it.

ps: some grammar may be incorrect although it tried hard.:p

With kind regards,
Guido

[trbarry]

I am not an expert on setting up web pages, so maybe I have this wrong. But it seems the problem is that you can detect unauthorized deep linking but if you refuse it then the dumb download managers just keep trying. Right?

Would it be possible to set it up so all detected unauthorized requests were satisfied by downloading a dummy module that, when run, just issued a sermon about good and bad download practices? That seems it would both get the message across and also satisfy the download managers so they wouldn't keep hammering on your door.

- Tom

[Koepi]

I'm sure i could modify the apache sources so that instead of a 404 - file not available or a 403 - acess forbidden _and_ "xvid" is in the filename, that i send a dummy file, but that's overkill.

There must be a more graceful way to do it.

(I added the referer check since those jerks that directly link to content don't update their links when a binary gets updated and thus the stupid leechers hammer on files which don't even exist...)

But thanks for the ideas.

If someone knows how to redirect special filenames with wildcards (and only in the case the file doesn't exist) to another file I'd be glad to implement that in my apache config.

Thanks for the help,
best regards
Koepi

[h0MBRe]

@koepi:

hmmm ... well, actually ... the purpose of providing this site of yours is not "having a site", but serving the public with fresh binaries.

so: why not use common p2p networks for distribution? this way you get rid of all the hassle, plus you save (your) bandwidth, because once you "inject" a new release via any kaazaa client (for instance), it should spread like a virus ;)

you might need to supply signatures (authentication hash values) of current releases to prevent alienating/corrupting your official distributions by "third parties", of course. i guess there are some tools out there, already, which provide this very functionality ;)

cheers,
h0MBRe

[trbarry]

Am I the only one who would be paranoid about binaries off p2p networks. At Nic's or Koepi's site I'm more confident I know where it's coming from.

- Tom

[Koepi]

No, I agree, p2p is no option (as i wrote above already ;) ).

Thanks for the idea though.

No apache experts around here? ;)

Best regards
Koepi

[h0MBRe]

i guess everyone who is capable of doing an encode with xvid is capable of opening a zipped file with winzip, choose "show column -> CRC", and compare the displayed crc32 value with the one supplied at the official page (i.e. kopies one).

actually, i do not know of any method to alter a file (or any data) while keeping the original crc32 hash value, so i consider this approach "safe".

also, for those resisting to use winzip (or equivalents), it might be easy but convenient to write a small command-line, cross-environment programm (let it be called kverify, "koepi's verify"), which retrieves the current crc32 value from a fixed location (http://some.server.net/dist/crc.xml), or a complete list of crc32 values of past distributions, via the system's default internet connection, and then compares it with the calculated crc32 value of a given (downloaded) file. perhaps a nice gui might come in handy ;)

just my two euro cents ...
h0MBRe

[wotef]

what do commercial pr0n sites do to stop hammers?

[duartix]

No apache experts around here?
Well this is the XVid forum, right?
Well Koepi it seems like your post here was a bit OT but I'm sure the moderator will close his eyes about that ;) :)

[Koepi]

Well duartix, if you think this is off-topic, I'm glad to present you with the alternative: no xvid binaries from me anymore.

Maybe you _might_ get a clue why it's related.

But I don expect that to happen to be honest.

:D :D :o

[cjv]

Koepi,

In your link, .../cgi-bin/fetch?file=XviD-24122002-1.exe what scripting language is "fetch" written in?
Think about how you serve the file.

It's been over a year since I've touched PHP or Perl, so I may be in over my head, but aren't you able to literally fopen and fwrite a file to stdout which then gets sent to the browser? Thus, get rid of the referrer check in apache entirely, so that anyone can access your XviD-ddmmyy-1 from anywhere.

Do the referrer check in PHP, and there you decide who is a valid referrer. If it's good, fopen your valid Xvid-ddmmyy-1.exe file and fwrite it to stdout (with a valid exe Application header). If the referrer is bad, just send the same filename, but fwrite a small message warning if they hammer you again, they get banned.

I really wish I could write something like this, but unfortunately I don't have the time needed to refresh myself with PHP...and then write the script. But, I'm sure someone in this forum is a scripting expert.
Just brainstorming here... :)

cjv

[Swede]

Well, I'm not an apache expert but I'm not sure that anything you do will get the wanted effect, since they're obviously to dumb to care. And I'm not sure if doing anything will lower your bandwidth since a 403 reply is very short. Anhow a short .htaccess like:
RewriteEngine on
RewriteCond /%{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} .*xvid.* [nc]
RewriteRule ^(.+) /leech.htm (or whatever)Will catch non-existant files that also contains xvid in the filename. I'm not really sure how this would interfere with your refcheck but that could also be done in this .htaccess. It will however put some extra load on you webserver since it has to check for the file first.

[Swede]

OK, had some more sparetime since my fever is keeping me from sleep :(

RewriteEngine on
RewriteCond /%{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} .*xvid.* [nc]
RewriteRule ^(.+) http://roeder.goe.net/whatever.exe[R,L]
RewriteCond %{HTTP_REFERER} !^http://roeder\.goe\.net/
RewriteRule \.(bz2|gz|pdf|zip|rar|exe)$ http://roeder.goe.net/leech.htm [R,L]This is based on you removing the cgi-referer-check. The first ruleset gives you the opportunity to redirect these 'offenders' to a URL of your likening. The second is the .htacces-referer-check that makes sure that every request for a *.zip, *.bz2, *.exe and so on, not originating from you gets redirected.

I'm still not conviced though that you will save any bandwidth but as always I might be wrong. Just the referer-check would be:
RewriteEngine on
RewriteCond %{HTTP_REFERER} !^http://roeder\.goe\.net/
RewriteRule \.(bz2|gz|pdf|zip|rar|exe)$ http://roeder.goe.net/leech.htm [R,L]
From a CPU/bandwidth-load point of view I would turn of the reverse-DNS lookup that you seem to have (from you log extracts). If you turn off HostnameLookups you can always postprocess the logfile using logresolve. (You might already do that, since the logs you've show might be from a postprocessed file but I just wanted to point this out.)

Oh, and since I'm not really fluent in Apacheish there might be errors in these scripts...

[Koepi]

Thanks for the input swede, it is very useful indeed!

I never thought about a "negative list" using the ! operator.

In fact, the script which is called does nothing more than increase a counter *giggle* amazing that people always think i'd write a referer check in perl ;)

I'm going to play around with that tomorrow, it is so useful that I can modify my apache config itself again to properly redirect to individual 403-pages.

And for the "dummy file": it'll be a few 100 bytes only (even smaller than my 403 page now which is ~5kb) and contains the text "die sucker" ;)

Ah, and btw: LOTR-TTT rules. I have to get some sleep now though.

Thanks for the valuable efforts!

Best regards
Koepi

[Koepi]

Hi again,

ok I implemented it now this way, created an anti-leech-page, compiled an xvid-dummy exe,... let's see how this works out :)

Thanks again a million,

best regards
Koepi

[Koepi]

Thanks for testing swede ;)

I remove the "file check" because now my router gets real heavy load (3.1 and rising). Wonder why that's so bad, those results should be chached somehow.
But finally, antileech seems to work fine at least! :)

Best regards
Koepi

[Swede]

If you examine your logs carefully you'll find another 'leeching' link, but I think you'll know which is mine :D
The anti-leeching works but a don't seem to get the dummy .exe, I get the Leeching detected even though I try do directly download a, now, non-existant version.

Edit: Hehe... Ok.

[Koepi]

Since the problem is gonna be resolved soon now I'll close this thread as we should concentrate on xvid itself again :)

Thanks everyone for the nice words, for the ideas and for the working solutions!

Best regards
Koepi