Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Announcements and Chat > Forum / Site Suggestions & Help

Reply
 
Thread Tools Search this Thread Display Modes
Old 16th July 2023, 00:37   #41  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,883
It's been a little over a month now, so I think that it was a good time to see where we're now.
Currently, Doom9 has a total of 153'952 topics and a little over 67'000 have been indexed.
This roughly accounts for 43.5% of the forum.
I can see lots of spiders scanning pages constantly including the Google ones, so it looks like it's just gonna be a matter of time before we get more topics back.
Unfortunately I didn't save those statistics before this whole thing happened, so I don't know where we were before in terms of indexing (I mean in 2022), but I think next month is gonna be a good time to get back and assess the situation again.

See you guys in one month.
FranceBB is offline   Reply With Quote
Old 25th July 2023, 04:56   #42  |  Link
Katie Boundary
Registered User
 
Katie Boundary's Avatar
 
Join Date: Jan 2015
Posts: 1,048
Solution: stop using Skynet products and services, and switch to the good guy team: Startpage, Qwant, Duckduckgo.
__________________
I ask unusual questions but always give proper thanks to those who give correct and useful answers.
Katie Boundary is offline   Reply With Quote
Old 25th July 2023, 09:14   #43  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
Quote:
Solution:
Not a solution for the many that do use google, and that is a problem for this site and not for you yourself (or anyone else) as an individual.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???
StainlessS is offline   Reply With Quote
Old 25th July 2023, 23:48   #44  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,883
Speaking of which, two or three days ago Doom9 suffered another outage for a few hours which I believe led to an IP address change, so Google stopped indexing again for a bit.
I thought Swede was gonna add another line to his logs but this time he didn't.
Speaking of which, it might be time to demand a new picture of the closet ehm I mean server room :P
FranceBB is offline   Reply With Quote
Old 30th September 2023, 03:45   #45  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,883
Quote:
Originally Posted by FranceBB View Post
It's been a little over a month now, so I think that it was a good time to see where we're now.
Currently, Doom9 has a total of 153'952 topics and a little over 67'000 have been indexed.
This roughly accounts for 43.5% of the forum.
I've been waiting a few more months and we're slowly but surely getting back to full index (although the sporadic downtimes don't make the process any easier).

Anyway, out of 153'952 we now have 79'200 indexed topics, so around 51.44% of the forum.
FranceBB is offline   Reply With Quote
Old 3rd March 2024, 18:04   #46  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,883
Hey everyone,
ever since the "incident", I keep looking at robots.txt https://forum.doom9.org/robots.txt
Looks like the new robots.txt is now blocking the entire website to the following spyders:

Byte (Chinese Search Engine), Ahrefs (a SEO company), Baidu (Chinese Search Engine) and Yandex (Russian Search Engine)

while Bing and Google are simply delayed so that they don't use too many resources while scanning the forum.
I use Google, so I don't personally care about Chinese and Russian search engines being blocked, but I'm just posting it here in case anyone does.
Still, I think we shouldn't block anything and rather use crawl-delay like we're doing on Google and Bing.

Speaking of Google, we're now at 83,900 pages there, up from 79'200 from 2023.

Last edited by FranceBB; 3rd March 2024 at 18:09.
FranceBB is offline   Reply With Quote
Old 3rd March 2024, 20:03   #47  |  Link
Swede
Deputy
 
Swede's Avatar
 
Join Date: Jan 2002
Location: Sthlm, Sweden
Posts: 1,451
That's correct. I've been monitoring the load the crawler puts on the server, which is quite heavy, and since Byte and Yandex doesn't honour the crawl-delay as they should (maybe no all but certainly a lot of them) I've just blocked them totally.
Swede is offline   Reply With Quote
Old 3rd March 2024, 23:44   #48  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,883
Ah, gotcha, makes sense.
Thanks as always for the clarification Swede.
FranceBB is offline   Reply With Quote
Old 4th March 2024, 13:44   #49  |  Link
Swede
Deputy
 
Swede's Avatar
 
Join Date: Jan 2002
Location: Sthlm, Sweden
Posts: 1,451
Finally put down some hours, found the worst culprits and IPbanned them. I've then reinstated all crawlers and quadrupled the RAM. This should give me some time to relax.
Swede is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 23:40.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.