Installed nginx as a proxy

[Swede]

I've installed nginx as a proxy since 'everyone' says that it's the best way to manage many connections.
Hopefully no one will notice any changes but if there's something broken drop a post here and I'll attend to it as quickly as possible (including reversing the entire thing if it's too bad).

/Swede

[Z2697]

What can possibly go wrong?

[kempodragon]

Famous last words :p

[Swede]

So far everything looks very good. Nothing seems to be broken and the load has gone down considerably! The servers didn't even blink at +60.000 users, not even the fans on the SQL-server started to spin(!)
I knew that many gurus recommend installing a nginx-proxy but I would never have guessed just *how* much it would help.

[Z2697]

I saw some database error earlier this day (my local time :)), is there something going on?

[StainlessS]

This was produced a little earlier today when I clicked on "Quick Links/Today's Posts".

I saved the image and a few moments later clicked "Today's Posts" again and no problems.
Thought I'de better post this anyway, in case it helps.

[clickme, twice]
https://i.postimg.cc/62Pb9DxS/Errors.jpg (https://postimg.cc/62Pb9DxS)

[Swede]

Yeah, sorry about that.. It was my fault. Note to myself: NEVER do three different updates at the same time again! :rolleyes:

Things should be smooth now though.

[Adub]

Hey Swede, thanks again for your continued hard work hosting the forum (and I've also downloaded the archive tar file as a backup, thanks for doing that).

I recently looked that the network requests, and I realized that the forum is still being served on HTTP 1.1. Have you thought about enabling support for HTTP2? Since you're using nginx already, it should be a pretty simple change: https://nginx.org/en/docs/http/ngx_http_v2_module.html

It has the potential to significantly reduce the number of connection requests, since HTTP2 multiplexes requests over a single TLS connection instead of opening multiple connections like HTTP 1.1.

It would also make the forum faster.

[Swede]

Hi Adub!

Thanks, I totally forgot about HTTP2 (and to enable it) when I installed nginx. It's now up and running.

/Swede

[Emulgator]

Good work !
(Maybe unrelated but as it just happened: I ran out of my monthly mobile hotspot quota, fell back to GPRS speed 64kbit/s, and the forum loads snappy as never before)

[Adub]

Awesome work, thanks Swede!

[Adub]

Hmm, I'm not sure if it's related to the HTTP2 changes (I doubt it) or if there were other changes made, but I'm getting "ERR_CONNECTION_CLOSED" errors when attempting to access the forum on an iPhone. I've tried several different internet connections.

Things work just fine on a laptop/desktop though, on the same internet connection.

Same thing happens on forum.doom9.se.

[Swede]

Oups, that was, obviously, a know bug with this setup (nginx, proxy, HTTP2, Safari or rather iOS).

Added:
# Fix iOS/Safari bug with HTTP2 behind proxy
proxy_hide_header Upgrade;

to the configfiles and things are ok... Again... For now...

/Swede

[Adub]

It's fixed now, thanks Swede!

[rwill]

Most users ever online was 154,577, Today at 02:54.

Some pages need a couple of seconds to load but the forum is still working overall...

[Swede]

Yeah, it's starting to be ridicuolus..

It seems that every optimization I do just invites more bots. Like two days ago, when I managed to *really* lighten to load on the SQL-server by adding another index, I happily went to sleep just to wake up to that new High Score of +150.000 :eek: And it seems that instead of cruising at ~20.000+ users the new normal is +60.000.

Well, at least I'm learning the art of optimizing. Maybe I'll explore the benefits of adding another SQL-server as my next step.

/Swede

[Jamaika]

Hey Swede, thanks again for your continued hard work hosting the forum (and I've also downloaded the archive tar file as a backup, thanks for doing that).

I recently looked that the network requests, and I realized that the forum is still being served on HTTP 1.1. Have you thought about enabling support for HTTP2? Since you're using nginx already, it should be a pretty simple change: https://nginx.org/en/docs/http/ngx_http_v2_module.html

It has the potential to significantly reduce the number of connection requests, since HTTP2 multiplexes requests over a single TLS connection instead of opening multiple connections like HTTP 1.1.

It would also make the forum faster.
As a layman, I don't know what or why, but I see a lot of excitement about http2. I see that http3 is already here.
The latest generation of the protocol, HTTP/3, brings further improvements, particularly in terms of speed and security. Google's solutions, specifically the QUIC protocol (HTTP/3 was initially called HTTP-over-QUIC), also played a significant role in this generation. The move away from the TCP protocol also proved crucial.
Google Gemini will begin to spy on you in full force, embedding Gemini in your messages, phone, and even WhatsApp. Theoretically, you can disable it, but there's practically no information about it.

No one seems to have any illusions anymore that Google or Apple are spying on us under the guise of "collecting information about our preferences." Now they're not hiding it, sending emails to their users and openly announcing that they'll start doing it in full force.

[Z2697]

As a layman, I don't know what or why, but I see a lot of excitement about http2. I see that http3 is already here.
The latest generation of the protocol, HTTP/3, brings further improvements, particularly in terms of speed and security. Google's solutions, specifically the QUIC protocol (HTTP/3 was initially called HTTP-over-QUIC), also played a significant role in this generation. The move away from the TCP protocol also proved crucial.
Google Gemini will begin to spy on you in full force, embedding Gemini in your messages, phone, and even WhatsApp. Theoretically, you can disable it, but there's practically no information about it.

No one seems to have any illusions anymore that Google or Apple are spying on us under the guise of "collecting information about our preferences." Now they're not hiding it, sending emails to their users and openly announcing that they'll start doing it in full force.

QUIC is just a fancy UDP, there's nothing that will make google spying effortless.
Sure, Chrome has most of the user base. But that's regardless of QUIC ot not.

[Z2697]

Yeah, it's starting to be ridicuolus..

It seems that every optimization I do just invites more bots. Like two days ago, when I managed to *really* lighten to load on the SQL-server by adding another index, I happily went to sleep just to wake up to that new High Score of +150.000 :eek: And it seems that instead of cruising at ~20.000+ users the new normal is +60.000.

Well, at least I'm learning the art of optimizing. Maybe I'll explore the benefits of adding another SQL-server as my next step.

/Swede

What was the record before the bot boom?

[Adub]

Yeah, it's starting to be ridicuolus..

It seems that every optimization I do just invites more bots. Like two days ago, when I managed to *really* lighten to load on the SQL-server by adding another index, I happily went to sleep just to wake up to that new High Score of +150.000 :eek: And it seems that instead of cruising at ~20.000+ users the new normal is +60.000.

Well, at least I'm learning the art of optimizing. Maybe I'll explore the benefits of adding another SQL-server as my next step.

/Swede

Do you have any rate limiting setup by chance? I know that nginx has native support for it, so I figure that would be a good preventative measure to prevent bot abuse.

[Swede]

The problem with Rate Limit is that the bots doesn't use the same IP, rather they span a /24 or even a /16 net so it's not much of a help.

I've been offline for a few days and the load was ridiculous when I returned kicking of another find-out-why-mission in the logs. There was an enormous increase in the 'Printable Version' of the threads, resulting in me removing that option. If you need/want that version pls use the archive as it's cached (and then of course sadly not 100% up to date but I'll try to update it somewhere like once a month).

[ChaosKing]

Another way would be to use nginx FastCGI Cache on the print version which bypasses php completely and serves static html after the first visit. But I don't think anyone will miss the print version.
btw same caching could be done for guests.

[Adub]

Honestly, I was starting to think about the caching for guests (not logged in) users as well. You can do that with a cookie presence check in your nginx config.

Even micro caching (1-10 seconds) could yield a massive benefit, depending on the load. Of course, go as high as is tolerable.

[Swede]

Normally I would agree but the loadproblem here would not benefit from caching since it's (almost) completely random.
I'm constantly monitoring the logs trying to find bottlenecks, like I'm now shooting down the try-to-reply-but-not-logged-in first thing in nginx so it wont generate any sql at all.

[MysteryX]

All the bots don't announce themselves as such? If it's AI agents, they should be identifying themselves.

Or is it more like a malicious attack trying to dodge detection?

[Adub]

That’s part of the problem - not all bots actually identify themselves. In fact, many intentionally mess with User Agent strings to attempt to hide themselves from detection.

If you’re thinking there’s an easy fix, think again. Swede has been working on this for a long time.

[Columbo]

CloudFlare is an easy fix for many other websites.

[MysteryX]

If Swede has been working on this for a long time, I'm surprised CloudFlare wasn't among the first fix attempts.

[MysteryX]

Currently active users: 3 members and 13477 guests

Something is definitely wrong here lol

Shouldn't need 20 CPUs to serve 3 members.

[Columbo]

If Swede has been working on this for a long time, I'm surprised CloudFlare wasn't among the first fix attempts. Apparently he's trying to avoid that. More power to him if he can sort things out local to his server(s).

Where's your donor badge? ;)

[Adub]

I’m also in favor of avoiding Cloudflare if at all possible. The centralization of the internet into the hands of the a smaller and smaller set of companies is very concerning imho.

I understand the reasoning behind the suggestion, but I think it should be an absolute last resort.

And I might suggest a competitor to Cloudflare -> bunny.net

Again, this is in hopes of preventing centralization, for a community that was literally built on the decentralized knowledge of many contributors.

[Z2697]

I’m also in favor of avoiding Cloudflare if at all possible. The centralization of the internet into the hands of the a smaller and smaller set of companies is very concerning imho.

I understand the reasoning behind the suggestion, but I think it should be an absolute last resort.

And I might suggest a competitor to Cloudflare -> bunny.net

Again, this is in hopes of preventing centralization, for a community that was literally built on the decentralized knowledge of many contributors.

Cloudflare is just a CDN though (but can still blow itself up and take half of the internet with it).
If hosting is still provided by Swede I'm fine calling it "not that centralizied".

[MysteryX]

Cloudflare is more than DNS, it has *lots* of features. Including shielding the server from DOS attacks; which is pretty much what's happening.

bunny.net is paid though.

Centralization of the internet in the hands of a few, good point. It's an optional dependency though, that can be removed if there's any issue.

Could test with it to see whether it manages to filter out the bad traffic, and perhaps give more insight. If it does, can then look at how else the same result can be achieved.

On Bunny website. There's a free plan on Bunny Shield.

HTTP Flood Protection
Not all attacks look like attacks. We stop them anyway.

Layer 7 attacks don’t scream—they whisper. Blending in with real traffic, they overwhelm your servers not with brute force but with deception. Bunny Shield’s HTTP flood protection sees through the noise. Using intelligent request analysis, behavioral fingerprinting, and adaptive mitigation, we stop malicious traffic before it ever reaches your origin. No CAPTCHAs. No latency. Just peace of mind, at any scale.

[Swede]

There are a couple of reasons I won't use Cloudflare, the dealbreaker being that they would be a 'Man-in-the-Middle' since they will decrypt all encrypted traffic between you and me which I consider as bad as reverting to HTTP-only.
Number two on my 'Wont-use-it'-list is that, as you might know, I'm not under any US law that requires me to share my data with any government whatsoever and that would then change since they can do whatever they want with the traffic. While this might not be of any huge significance to you, it is to me. It's a question of trust. (And as long as I manage to keep the forum up most of the time why should I release control to an unknown number of others)

Then there is the cost, I don't think we can use the Free plan with +200 million requests / month (I guess they do count all requests not just the ones they let through).

Oh, and this aren't DDOS attacks, it's just a %&#¤ of a lot of bots scraping the net.

(And don't get me wrong, if you want to use Cloudflare it's up to you, I don't have any problems regarding those kind of decisions. This is just my way)

[GeoffreyA]

Would Anubis be any better?

[Z2697]

It's sad that whatever you do (or don't do) to somehow distinguish the requests, is gonna impact real user experience.
This is what modern internet has become...

[GeoffreyA]

Sadly, and will continue for the foreseeable future.

[wonkey_monkey]

Layer 7 attacks don’t scream—they whisper. Blending in with real traffic, they overwhelm your servers not with brute force but with deception. Bunny Shield’s HTTP flood protection sees through the noise. Using intelligent request analysis, behavioral fingerprinting, and adaptive mitigation, we stop malicious traffic before it ever reaches your origin. No CAPTCHAs. No latency. Just peace of mind, at any scale.

I don't know about anyone else but this obviously AI-generated marketing dreck is a red flag to me...

Would something as simple as a delay in loading pages - just a few seconds, if you don't have a "logged in" cookie - be enough to put a few bots off, plus a return to default 20 posts per page (or less), without inconveniencing real guests too much? It would just need a tactical sleep(...); deployed in the right .php file.

I don't suppose it's as simple as seeing an unusually high number of requests from just one IP address or block...

[Swede]

Sadly Anubis is not an option, for the same reason 'high number of requests from just one IP address' isn't. >90% of the requests are from different IPs (just one req / IP) so the amount of CPU to either challenge (Anubis) or blocking IPs (fail2ban) are no good.
And the 20 post / page would actually increase the load since the overhead of splitting one (large) thread into +1000 pages drains both the SQLserver and the web-server. I've done quite some trial-and-error with this.

Sadly, as you've concluded, we probably have to live with this for a while, there are too many clients out there for anyone to have hardware that manages (for a reasonable cost).
Just look at the latest 'HighScore': Most users ever online was 170,558, 17th September 2026 at 23:04. (GMT+2)

[Z2697]

What are they doing though, just browsing?

[wonkey_monkey]

And the 20 post / page would actually increase the load since the overhead of splitting one (large) thread into +1000 pages drains both the SQLserver and the web-server.

I thought (as in wondered) that if combined with a sleep it might lighten the load overall. Instead of 1x 1000-post request being fulfilled immediately, you'd be spreading 50x 20-post requests across, say, 100 seconds (do sequential requests for a thread's pages come from the same IP?). If the spurious requests tend to be on threads of all ages, you could increase the sleep time on old threads.

[Swede]

I've turned some knobs, flipped some switches and added an overwatch script that I hope will help.

[MysteryX]

Still -- it could be useful to know whether CloudFlare or Bunny is able to filter out the traffic or not.

If yes, then there's a way to filter out the traffic. It needs to be filtered out *before* reaching the message board. Maybe nginx proxy can integrate such a filter.

If it reaches the forum and you try to fight it from there, it's like Jujitsu: if you start on the floor, you already lost.

I'd start with the advanced solutions out there like CloudFlare; if they have a good enough algorithm to filter it out, then I'd look at how to implement such filter within nginx.

I asked on my AI stack


Nginx alone

A filter on nginx can help, but not the filter people reach for first.

Per-IP `limit_req` does almost nothing here. One request per IP never trips a rate. UA maps miss anything that says `Mozilla/5.0 … Chrome`. `robots.txt` is already ignored. Blocking Alibaba-class ASNs is already what he does; the rest is residential.

What actually works on the origin, without handing the domain to Cloudflare:

1. Stop writing a guest session for anonymous hits. The SMF crowd documented the same wound: every bot INSERT/UPDATE on `log_online` is the real load. If vBulletin can be told not to persist guests, the counter and the SQL storm both collapse. That is the highest-leverage change and it lives in the app, not nginx.
2. Cache anonymous forum pages at nginx (`proxy_cache` on GET for guests, short TTL, bypass for logged-in cookies). Bots then hit disk/RAM, not PHP.
3. Put a self-hosted challenge in front of PHP. Anubis sits between nginx and the backend, issues a proof-of-work / JS check, sets a cookie, and only then proxies. No third-party decrypts the public traffic. Forum members already mentioned it in the December load thread. It is the closest thing to Cloudflare’s challenge without leaving the machine.
4. Keep the existing UA + datacenter ASN denylist as a cheap first cut. Use `return 444` so nginx closes without a response body.

If he will not run a challenge and will not change guest-session behavior, nginx can only absorb, not filter. The IPs have no pattern because that is how this botnet is built. Edge reputation and a browser proof are what distinguish “one hit from a toaster in Mumbai” from a person. Without that signal, origin nginx is just a faster door.

[FranceBB]

Shouldn't need 20 CPUs

To be fair, during the last wave of bots, I was closely monitoring the load and the server itself was at about 40% CPU load, which is why the forum itself - albeit displaying the "busy" message and not allowing you to browse the pages - was indeed reachable.

The problem seems to be the database not being able to cope with the requests.

I've turned some knobs, flipped some switches and added an overwatch script that I hope will help.

*fingers crossed*

In Swede we trust.

[MysteryX]

The problem seems to be the database not being able to cope with the requests.
So this is definitely database writes. If guest visits no longer write log to the database, it would reduce the load massively.

[Swede]

To end all speculations, the problem now is not the database. The sql-server is barely humming these days. The problem is that when the bursts come, they are massive. They are all legitimate requests, it's just that there are too many. I won't devulge how but I do believe I can now make nginx drop most of them but only time will tell. Now I'll concentrate on tuning the balance between the two servers.

It's not that I don't appreciate you all trying to help but just guessing without all the facts (logs and so on) is of little use.

For those of you that knows your logfiles, these examples are to show you the sheer size (rotation every day):
-rw-rw-r-- 1 nginx apache 1.8G Sep 20 00:12 forum.doom9.access_log-20260920
-rw-r----- 1 nginx root 1.1G Sep 20 00:15 forbidden.doom9.log-20260920

[Adub]

To end all speculations, the problem now is not the database. The sql-server is barely humming these days. The problem is that when the bursts come, they are massive. They are all legitimate requests, it's just that there are too many. I won't devulge how but I do believe I can now make nginx drop most of them but only time will tell. Now I'll concentrate on tuning the balance between the two servers.

Just gotta say - you worked some magic, Swede. Forum has been rock solid for me this week+. Glad you could get things stabilized with some nginx foo.

[FranceBB]

Yep, whatever you did worked great. I can see it cruising at around 20k connections all the time and it's very stable. Very well done! :D