Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
16th July 2023, 00:37 | #41 | Link |
Broadcast Encoder
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,883
|
It's been a little over a month now, so I think that it was a good time to see where we're now.
Currently, Doom9 has a total of 153'952 topics and a little over 67'000 have been indexed. This roughly accounts for 43.5% of the forum. I can see lots of spiders scanning pages constantly including the Google ones, so it looks like it's just gonna be a matter of time before we get more topics back. Unfortunately I didn't save those statistics before this whole thing happened, so I don't know where we were before in terms of indexing (I mean in 2022), but I think next month is gonna be a good time to get back and assess the situation again. See you guys in one month. |
25th July 2023, 04:56 | #42 | Link |
Registered User
Join Date: Jan 2015
Posts: 1,048
|
Solution: stop using Skynet products and services, and switch to the good guy team: Startpage, Qwant, Duckduckgo.
__________________
I ask unusual questions but always give proper thanks to those who give correct and useful answers. |
25th July 2023, 09:14 | #43 | Link | |
HeartlessS Usurer
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
|
Quote:
__________________
I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ??? |
|
25th July 2023, 23:48 | #44 | Link |
Broadcast Encoder
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,883
|
Speaking of which, two or three days ago Doom9 suffered another outage for a few hours which I believe led to an IP address change, so Google stopped indexing again for a bit.
I thought Swede was gonna add another line to his logs but this time he didn't. Speaking of which, it might be time to demand a new picture of the closet ehm I mean server room :P |
30th September 2023, 03:45 | #45 | Link | |
Broadcast Encoder
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,883
|
Quote:
Anyway, out of 153'952 we now have 79'200 indexed topics, so around 51.44% of the forum. |
|
3rd March 2024, 18:04 | #46 | Link |
Broadcast Encoder
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,883
|
Hey everyone,
ever since the "incident", I keep looking at robots.txt https://forum.doom9.org/robots.txt Looks like the new robots.txt is now blocking the entire website to the following spyders: Byte (Chinese Search Engine), Ahrefs (a SEO company), Baidu (Chinese Search Engine) and Yandex (Russian Search Engine) while Bing and Google are simply delayed so that they don't use too many resources while scanning the forum. I use Google, so I don't personally care about Chinese and Russian search engines being blocked, but I'm just posting it here in case anyone does. Still, I think we shouldn't block anything and rather use crawl-delay like we're doing on Google and Bing. Speaking of Google, we're now at 83,900 pages there, up from 79'200 from 2023. Last edited by FranceBB; 3rd March 2024 at 18:09. |
3rd March 2024, 20:03 | #47 | Link |
Deputy
Join Date: Jan 2002
Location: Sthlm, Sweden
Posts: 1,451
|
That's correct. I've been monitoring the load the crawler puts on the server, which is quite heavy, and since Byte and Yandex doesn't honour the crawl-delay as they should (maybe no all but certainly a lot of them) I've just blocked them totally.
|
Thread Tools | Search this Thread |
Display Modes | |
|
|