Recent forumload

[Swede]

There has been an incredible increase in guests (anonymous accounts and bots) recently! The number of guests used to be something like a thousand or so with peeks at ~2000 but now it's more like +12.000 with the extreme of >32.000 the other day.
Well, since my server couldn't cope (of course) I had to add a dedicated sql-server to my collection of things that costs me not only time and purchase but also in electricity.
For now things are looking ok, both servers are humming, the internetcables are glowing and the temperature is rising.

So, for the first time in 20 years, I'm setting up a PayPal donation account (forum@doom9.org) so anyone that feels they want to contribute to the cost of running this can send me a couple of $ or so. :cool:

Anyway, I'll be keeping a close eye on the load for a couple of days (until santa arrives).

Rgds,
Swede

[Columbo]

How did you mitigate the guest/bot overload?

[Swede]

I moved the sql-database to a new, dedicated server... The old one that used to do everything is now 'just' a webserver and the new 8 core i7-7700T takes care of all sql-stuff. So far it seems to be powerfull enough.

[Sunspark]

Are these actual viewers or bot crawlers or some kind of weird DoS?

[microchip8]

These are AI scrapers most likely. Many busy forums and git repos have been suffering from them, some even implementing things like Anubis browser checks https://anubis.techaro.lol/

[Swede]

I'm sure it's not real users. The main part now is probably just someone with a botnet wishing us harm. There are a lot of AI-bots nowadays but not this many.

[Columbo]

So you upped your server juice and just let the bots do as they please? Forums I am associated with use Cloudflare with a little extra pixie dust to cut them off at the pass.

[Columbo]

I just sent you $50 to get the telethon started. Happy to help and thank you for all you do.

[GeoffreyA]

These are AI scrapers most likely. Many busy forums and git repos have been suffering from them, some even implementing things like Anubis browser checks https://anubis.techaro.lol/

Yes, the Anubis checks are increasingly used of late, as these things get worse.

[SirMaster]

What about just putting the site behind Cloudflare like most people do these days?

[kurkosdr]

These are AI scrapers most likely. Many busy forums and git repos have been suffering from them, some even implementing things like Anubis browser checks https://anubis.techaro.lol/
The problem with Anubis is that it makes the website not indexable by search engines, so you have to manually exclude the IPs of the search engine crawlers so they aren't hit by an Anubis challenge. Does such a public list of search engine crawler IPs exist?

[Swede]

..and it looks like we're back online. That was a *huge* DDos-attack. (Going to try Cloudfare)

[Z2697]

What a harsh environment...

[Jamaika]

These are AI scrapers most likely. Many busy forums and git repos have been suffering from them, some even implementing things like Anubis browser checks https://anubis.techaro.lol/
I'm not familiar with forum congestion. If I specify the page playback speed of the forum.doom9.org, will the forum crash?
https://tools.pingdom.com/

[Jamaika]

There has been an incredible increase in guests (anonymous accounts and bots) recently! The number of guests used to be something like a thousand or so with peeks at ~2000 but now it's more like +12.000 with the extreme of >32.000 the other day.

I'm guessing 2,000 views now. It might be 15,000 views of something unspecified during the holidays. ;)

[Jamaika]

Strangely enough, the Programming and Hacking thread is currently invisible to non-logged-in users. Was I to blame? :D

[Swede]

Hehe, no.. That was just me forgetting to reopen that section. :rolleyes:

[Z2697]

What does it cost to run the Doom9 forum?

[real.finder]

I cant browse the doom9 without a VPN since few days, my ISP IPs is banned or something?

[Swede]

Well, since you're using a VPN I cant check your real address but drop me a PM and I'll check.

[real.finder]

Well, since you're using a VPN I cant check your real address but drop me a PM and I'll check.

done, and now seems I can see doom9 without a VPN :thanks:

[FranceBB]

I'm sorry to hear that, but I'm glad to see that the new hardware seems to be coping well with the new number of users.
There are currently 10483 users and the forum is loading as fast as ever. I know it got upgraded "for the wrong reasons" (i.e bots) but this hardware upgrade definitely helped regular users too.
Stats look fine too.

23:56:43 up 3 days, 10 min, 6 users, load average: 2.02, 1.87, 1.80

total used free shared buff/cache available
Mem: 32740824 7626948 12690544 69888 12955544 25113876
Swap: 24948728 8803528 16145200

Nof http-processes: 10

php: 78

Forumclients: 11610


Now I'm curious to see the new picture of the closet :D

[Blue_MiSfit]

Donation en route

[Adub]

Not sure what the software stack is like, but running some caching software like Varnish/Vinyl (https://vinyl-cache.org/) can help with managing load.

In the name of competition, an alternative to CloudFlare is Bunny: https://bunny.net/shield/

[filler56789]

I cant browse the doom9 without a VPN since few days, my ISP IPs is banned or something?

Same here, since, let's say, 24 hours ago.
IP address from Brazil.

[Swede]

Yeah, sorry about that. I had to shut down some IPseries in Brazil for a while, should be back now.

[takla]

The server is too busy at the moment. Please try again later.
I still get a ton of these when I want to load a page.