Bug - "The Server is too busy" but actually it's not

[FranceBB]

Hi there guys,
every once in a while I'm getting the following message:

vBulletin Message
The server is too busy at the moment. Please try again later.


https://i.imgur.com/pqOA0dF.png

and the forum stats say:

Currently Active Users: 3465 (9 members and 3456 guests)

which looks absolutely fine given that we peaked at 17k in 2017 without the blink of an eye.
When I went to check the actual server workload here (https://www.leffster.se/) the stats were also pretty normal:

19:04:23 up 26 days, 10:03

load average: 13.50, 25.75, 20.01

Memory:
total 8015288
used 3466960
free 3581488
shared 30580
buff/cache 1313160
available 4548328


Swap:
total 16186360
used 3013412
free 13172948



So... I'm a bit puzzled...
Is it vBulletin being dodgy?
Are the stats not actually right?
Dunno... but I'd like to know if it happened to other people too.

[Selur]

Getting "The server is too busy at the moment. Please try again later. " more and more often the last few days.
Maybe lowering "Cookies and HTTP Header Options -> Session Timeout" might help,... (I suspect the board is running into some mqsql or similar session limitation limits)

which looks absolutely fine given that we peaked at 17k in 2017
may be the session time out then simply was larger ? (lager time-outs => more users)

[manolito]

Starting this morning all these issues have disappeared... :D

A big THANKS to anyone who reported it and to Swede who was probably the one who fixed it.


Thanks again
manolito

[FranceBB]

It is much smoother now, but then again I've been checking the stats and we've always been in the range of 1800 users connected, down from the 3500 peak of the other days.
Perhaps some of those were not actually users and were DDOS attacks? Dunno. Still, it's much better now, so thank you Swede! :)

[manolito]

Hi FranceBB,

it looks like my last lost came a little too early. After 1 day of almost normal speed the Doom9 site is very slow again. So far no messages like "The server is too busy at the moment. Please try again later.", but the response time when switching to a different forum is very slow. Something is definitely broken here... :scared:

Cheers
manolito

[FranceBB]

To be fair, the reason why it was snappy again when you and I both posted was that we were in the 1800 concurrent users range, however now we're back to 3300-3800 concurrent users range and we're seeing the same "issue" as before.
Those seem however to be legitimate users, so I guess it's just the fact that Doom9 is popular.
Given that the server has been up and running for 34 days straight, perhaps Swede could try the oldest trick in the world: rebooting it. Dunno.
We're currently at 3259 concurrent users and the server usage looks like this:

up 34 days load average: 14.30


Memory
Total: 8015288
Used: 4075088
Free: 1678064
Shared: 52696
Buff/Cache: 2609208
Available: 3940200

Swap
Total: 16186360
Used: 2936800
Free: 13249560

which doesn't look terribly bad, though, so perhaps it's not the server but rather his ISP capping the bandwidth?

[manolito]

You may be correct, but what puzzles me is that the slowdown did not come gradually, it came suddenly which looks a lot like some change in the board software, or the ISP which hosts the board changed something in its server configuration.

Whatever, it does look a bit suspicious to me, I hope it is not a sign of worse things ahead... :scared:


Cheers
manolito

[nevcairiel]

A load average of 14 or even 25 in the OP is in fact rather bad.
Right now its at 2 with 1800 users, thats more in the range of sensible. Twice the users should not explode that to 7-10 times as high load, unless there is a weird bottleneck somewhere.

[Swede]

Hi guys, sorry for being offline for some days... I haven't done anything in the server that should affect the performance. I've now updated the OS and rebooted. Will keep an eye open.

[manolito]

Hello Swede,

whatever you did, it sure did the trick... (at least for me) :thanks:
Speed is back to what it was before.


Thanks again...
manolito

[FranceBB]

It might be the oldest trick in the world, but if I learned one thing during all those years is that rebooting always works.
This was one more confirmation of it. ;)
Thanks, Swede.

[m0rbidini]

Got no errors in the last couple of days! Thanks.

[rwill]

Aaaand... its busy again.

**
13:37:31 up 5:10, 2 users, load average: 14.43, 13.67, 14.09

[FranceBB]

Well, there were almost 5000 people both today and yesterday. Even in this very moment there are over 3000.
They don't seem to be bots as the AI training bots scraping the web are all blocked in robots.txt as you can see here: https://forum.doom9.org/robots.txt

robots.txt - as configured by Swede - is already blocking Amazonbot, Bytespider, SemanticScholarBot, PetalBot, YandexBot, GPTBot, DotBot, SemrushBot and ClaudeBot.

Whether those 5000-3000 people are actual genuine people or just bots lying on their user agents I don't know.


EDIT:
It actually reached 7556 users at 21.21 BST. The load reported is 25.72% though in here: Link (https://www.leffster.se/)

Resource usage looks ok though:

Memory
Total: 32745864
Used: 8386328
Free: 22984952

So if I'm reading it correctly, out of 32GB of RAM, only 8GB were used and 22GB were free.
So... Is the load affecting the CPU instead? Perhaps the bandwidth? Dunno. Only Swede knows.

[filler56789]

Aaaand... its busy again.

Yes. And when it was not busy, it was... whatever :-/

https://forum.videohelp.com/attachments/81410-1723409523/doom9unstable.png

[rwill]

EDIT:
It actually reached 7556 users at 21.21 BST. The load reported is 25.72% though in here: Link (https://www.leffster.se/)

Resource usage looks ok though:

Memory
Total: 32745864
Used: 8386328
Free: 22984952

So if I'm reading it correctly, out of 32GB of RAM, only 8GB were used and 22GB were free.
So... Is the load affecting the CPU instead? Perhaps the bandwidth? Dunno. Only Swede knows.

You seem to have a misconception about what "load" under Unix is. For most Unix Systems its the process queue length, so a load of 25 does not mean 25% but that there are 25 processes that can do work at the same time. You would need around 25 CPU cores, regardless of speed, to handle this load. There are many pages on the web explaining this.

I do not know what kind of system Doom9 runs on, for a 32 core system a load of 25 is maybe still fine, for a 8 core system, maybe not if the process requests stack.