FranceBB
15th October 2023, 06:41
Hi there,
first of all kudos to Swede for keeping this wonderful forum alive for over 20 years, we all appreciate this very long time commitment.
A while ago the conversation about Google indexing and robots.txt came up: https://forum.doom9.org/showthread.php?t=184892
To recap the topic for those who didn't follow it, Doom9 was being targeted by spambots and other attacks flooding it. Not only this was annoying for users who kept seeing spam posts, but being the nature of the whole site based on self hosting in Swede's closet https://www.leffster.se/ with DDOS they managed to bring it down a few times, causing outages.
To avoid this, robots.txt ban on several user agents was introduced in December 2022, but unfortunately one of them was Google which made all the pages disappear.
StainlessS reported the issue earlier this year and the robots.txt was removed to allow Google to index pages again.
Not only we lost our pagerank, but to this day Google hasn't finished reindexing all the forum pages yet.
On top of that, attacks started again on Doom9 and this isn't great either, which is why the robots.txt was brought back http://forum.doom9.net/robots.txt but unfortunately this means that we're back to where we started.
As a free and safe solution to this, I think going behind cloudflare should be considered.
I myself host a website on OVH and I've been doing that since June 2006.
Ever since I put it behind cloudflare's free tier a year ago all my problems disappeared.
Not only it saved a lot of bandwidth, but it also actively protected my website from several malicious attacks and I can see all the statistics in my monthly reports (I'll post them later).
In other words, I do think that the whole website should be put behind cloudflare.
What do you think? Swede?
EDIT: Cloudflare's report for my website
Cloudflare served 406.73 GB of data, and mitigated firewall events this month.
Top Traffic Locations
Italy: 1,725,768
United States: 198,269
Germany: 74,430
France: 50,966
Other: 294,717
FIREWALL MITIGATED EVENTS 227
first of all kudos to Swede for keeping this wonderful forum alive for over 20 years, we all appreciate this very long time commitment.
A while ago the conversation about Google indexing and robots.txt came up: https://forum.doom9.org/showthread.php?t=184892
To recap the topic for those who didn't follow it, Doom9 was being targeted by spambots and other attacks flooding it. Not only this was annoying for users who kept seeing spam posts, but being the nature of the whole site based on self hosting in Swede's closet https://www.leffster.se/ with DDOS they managed to bring it down a few times, causing outages.
To avoid this, robots.txt ban on several user agents was introduced in December 2022, but unfortunately one of them was Google which made all the pages disappear.
StainlessS reported the issue earlier this year and the robots.txt was removed to allow Google to index pages again.
Not only we lost our pagerank, but to this day Google hasn't finished reindexing all the forum pages yet.
On top of that, attacks started again on Doom9 and this isn't great either, which is why the robots.txt was brought back http://forum.doom9.net/robots.txt but unfortunately this means that we're back to where we started.
As a free and safe solution to this, I think going behind cloudflare should be considered.
I myself host a website on OVH and I've been doing that since June 2006.
Ever since I put it behind cloudflare's free tier a year ago all my problems disappeared.
Not only it saved a lot of bandwidth, but it also actively protected my website from several malicious attacks and I can see all the statistics in my monthly reports (I'll post them later).
In other words, I do think that the whole website should be put behind cloudflare.
What do you think? Swede?
EDIT: Cloudflare's report for my website
Cloudflare served 406.73 GB of data, and mitigated firewall events this month.
Top Traffic Locations
Italy: 1,725,768
United States: 198,269
Germany: 74,430
France: 50,966
Other: 294,717
FIREWALL MITIGATED EVENTS 227