Google no longer searches forum

[StainlessS]

For some reason, Google no longer returns results of forum search on eg

avisynth site:forum.doom9.org

https://www.google.co.uk/search?q=avisynth++site%3Aforum.doom9.org

or

avisynth site:https://forum.doom9.org

https://www.google.co.uk/search?q=avisynth++site%3Ahttps%3A%2F%2Fforum.doom9.org

Is it problem with Google, or with forum.

https://i.postimg.cc/bGdq3fj1/Untitled-00.jpg (https://postimg.cc/bGdq3fj1)

EDIT:
This works,

ukraine site:bbc.co.uk

https://www.google.co.uk/search?q=ukraine+site%3Abbc.co.uk

EDIT: Perhaps forum has added some kind of "robots" thingy, preventing scanning of site.

[Thundik81]

https://forum.doom9.org/robots.txt
https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt

[StainlessS]

Thanks for that, Thundik81. [EDIT: I forgot how that robots thingy works, but remembered was 'robots' related]
It would then seem that GoogleBot has been denied access to search the D9, one wonders why that is.
Presumably then, you will never get any Doom9 forum hits when doing a google search.

Planned obsolescence ???
(If you are not already a Doom9 user, you will never [well hardly ever] find out that it exists)

EDIT: "Yandex Spider" and "BingBot" are currently scanning forum. (See Quick Links/Who's online).
Why the hatin' on Google, but not Yandex nor Bing ?

[cubicibo]

I was also attempting to use the Google Search Engine to search the forum with short keyword and had the same issue. I don't think this robots.txt change is good.

[FranceBB]

I always use google to search the forum.
Not only Google used to provide far better search results than the internal search inside the forum, but it was also how the overall majority of users came to Doom9 in the first place.
They searched something on the web related to encoding, Google provided topics in the forum as answers, they opened the related topics, found the replies and said "you know what? I'm gonna stick around, those people know what they're talking about".

Blocking Google from scanning the forum is a big mistake and will inevitably lead to a sharp fall in viewers/guests and an inevitable decline in new users...


p.s even the .net is blocking Google:


avisynth site:forum.doom9.net


returns nothing

[Balling]

Thanks for that, Thundik81. [EDIT: I forgot how that robots thingy works, but remembered was 'robots' related]
It would then seem that GoogleBot has been denied access to search the D9, one wonders why that is.


http://forum.doom9.net/robots.txt

[StainlessS]

Yeah I got that Balling. [Thundik81 already posted where to find it]
one wonders why that is.
Meaning, Why was Google denied access by D9 glitterati.

[Selur]

When I do a google search against "avisynth site:forum.doom9.net" I see the info that results were removed because of some DMCA claim,... which points to https://www.lumendatabase.org/notices/10021144

[filler56789]

When I do a google search against "avisynth site:forum.doom9.net" I see the info that results were removed because of some DMCA claim,... which points to https://www.lumendatabase.org/notices/10021144

Thanks for the useful info👍, Selur.
Now we understand why Yandex was not blocked by the forum admin(s):
Gluegow is a slave to the c0pyrite wh0r&$, but Yandex is not.😇

[filler56789]

When I do a google search against "avisynth site:forum.doom9.net" I see the info that results were removed because of some DMCA claim,... which points to https://www.lumendatabase.org/notices/10021144

What browser were you using?
With Opera version 93.etc, both "avisynth site:forum.doom9.net" and "avisynth site:forum.doom9.org" return that mega-stupid mega-ugly mega-annoying fishertroll animated GIF🤮

https://forum.videohelp.com/threads/405805-More-proof-that-Google-is-designed-by-DD00TS

[nevcairiel]

When I do a google search against "avisynth site:forum.doom9.net" I see the info that results were removed because of some DMCA claim,... which points to https://www.lumendatabase.org/notices/10021144

That claim is 9 years old. Either indexing was broken for all this time and you never noticed, or its not related.

[StainlessS]

I last searched @ site:forum.doom9.org [.org only] about December, worked back then. [EDIT: I was inactive about Jan->March]

That claim is 9 years old. Either indexing was broken for all this time and you never noticed, or its not related.
Astute observation.

EDIT:

With Opera version 93.etc, both "avisynth site:forum.doom9.net" and "avisynth site:forum.doom9.org" return that mega-stupid mega-ugly mega-annoying fishertroll animated GIF

I'm using FireFox, and for the .net search, I could not see the DMCA message as was off the bottom of the page, required scrolling to see it.
(W10 settings/System/Display/Scale & Layout=150%, ALSO, I got FireFox at Zoom 150%, 4K monitor)
EDIT: And I love my SUPER MASSIVE <requires about:config editing> scroll bars, so easy to hit [clickme, twice].
https://i.postimg.cc/c6JxTQYM/Untitled-00.jpg (https://postimg.cc/c6JxTQYM)

[Thundik81]

I last searched @ site:forum.doom9.org [.org only] about December, worked back then. [EDIT: I was inactive about Jan->March]

Wayback Machine is your friend:
http://web.archive.org/web/*/https://forum.doom9.org/robots.txt
>
http://web.archive.org/web/20221224004844/https://forum.doom9.org/robots.txt
then Santa Claus came to Doom9
http://web.archive.org/web/20221228033803/http://forum.doom9.org/robots.txt

[VoodooFX]

As I understand doom9's robot.txt doesn't disallow Google searching the forum threads.

Strange that it doesn't show result from doom9, DMCA complains removes one link, not a whole site. Last DMCA I see was in 2021.

duckduckgo works, if I remember correctly they actually are using data from Google: https://duckduckgo.com/?q=avisynth+site%3Aforum.doom9.org&t=h_&ia=web

[StainlessS]

Thanx again Thundik81,
(And I just found out that my ISP is no longer blocking Wayback Machine.)

[Brazil2]

duckduckgo works, if I remember correctly they actually are using data from Google
AFAIK data is from Microsoft Bing:
https://spreadprivacy.com/more-privacy-and-transparency/
...our use of Bing as a source for our private search results

Using 'netstat -ano' or TcpView (https://learn.microsoft.com/en-us/sysinternals/downloads/tcpview) you can also see that Duckduckgo connects to Microsoft IP addresses when you perform a search.

[Swede]

I don't know why google have banned us but I'm quite sure that disallowing "User-agent: Googlebot/Nutch-1.7" is not the problem (you can google it :)). Just to be sure I've removed that restriction and will monitor the abuse of Nutch that we were hit with at that time as well as looking into why google doesn't like us anymore.

[gonca]

Could it be a search algorithm update?
They had one in the second half of March

[Balling]

I don't know why google have banned us but I'm quite sure that disallowing "User-agent: Googlebot/Nutch-1.7" is not the problem (you can google it :)). Just to be sure I've removed that restriction and will monitor the abuse of Nutch that we were hit with at that time as well as looking into why google doesn't like us anymore.

I noticed that every time I searched previously it was actually the print pages not forum pages and only from .net mirror, that has no valid https cert, because the cert has no .net alternate name.

[filler56789]

I noticed that every time I searched previously it was actually the print pages not forum pages and only from .net mirror, that has no valid https cert, because the cert has no .net alternate name.

MOSTLY from the "printer-friendly :rolleyes: archive", yes, but not always.
And in the better old days, G00gle also indexed the "semi-official" Doom9 mirrors by Koepi and Nic.

[FranceBB]

And... we're back, baby.
Google slowly started indexing pages again.
Not only I could see the spider, but if you click on StainlessS first link (or indeed search avisynth site:forum.doom9.org yourself on Google) you'll see that we now have 1 topic:
"The power of Avisynth: restoring old 8mm films."

I wonder how long it will take for Google to index back all the thousands of topics in the forum and if we can do something to speed things up...
Perhaps setting up an account with the Google Webmaster Tool would help? https://search.google.com/search-console/about

[StainlessS]

We're gettin' a few hits on

avisynth+ site:forum.doom9.org

https://www.google.co.uk/search?q=avisynth%2B++site%3Aforum.doom9.org
but still currently only one hit on 'avisynth' (non +) search alone.

Most, (but not all) 'avisynth+' hits have 'avisynth+' in the thread title. (not searching body text ???)

[filler56789]

We're gettin' a few hits on

avisynth+ site:forum.doom9.org

https://www.google.co.uk/search?q=avisynth%2B++site%3Aforum.doom9.org
but still currently only one hit on 'avisynth' (non +) search alone.

Most, (but not all) 'avisynth+' hits have 'avisynth+' in the thread title. (not searching body text ???)

Since 2013 at least, Google's databases are programmed to become more and more "selective"
(read: self-corrupted) :-|

[StainlessS]

Maybe its now due to AI being more selective, and the thread
"The power of Avisynth: restoring old 8mm films.", is the only one that
it's AI deems worthy [ you watchin' fred ] :)

[FranceBB]

Nah, I think it's just gonna be a matter of time.

Searching with link (https://www.google.co.uk/search?q=site%3Aforum.doom9.org&sxsrf=APwXEddxiFgYHh9_tFGzc4yHrMA0TWLVaQ%3A1684952401925&ei=UVVuZIz8N8-GhbIPzZqpiA0&oq=site%3Aforum.doom9.org&gs_lcp=ChNtb2JpbGUtZ3dzLXdpei1zZXJwEANKBAhBGABQkiVYjSlgyC1oAHAAeAKAAQCIAQCSAQCYAQCgAQHAAQE) shows that we're already getting lots of pages back, but it looks like it's gonna take a while before the whole forum is indexed back.

I'm afraid it's not gonna be all laugh and smiles, though, 'cause although pages are back now that the ban is gone, Google is really not ranking Doom9 highly, so much so that even searching Avisynth doesn't show Doom9 in the first page and we're probably somewhere further down in Google, like in some remote high page number.
I really hope I'm wrong and I know that the admins were trying their best, but it looks like the effects of this self inflicted penalty (by mistake) will last for quite some time... :(

[StainlessS]

Today at time of posting, showing (for "site:forum.doom9.org/")
About 29,200 results (0.14 seconds)

Whats it gonna say tomorrow. [EDIT: Ie number of results]

[filler56789]

Today at time of posting, showing (for "site:forum.doom9.org/")
About 29,200 results (0.14 seconds)
Whats it gonna say tomorrow.

Doesn't matter. Because "About NNNN results (0.nn seconds)" depends on which part of the planet you're in, to begin with :-/

Besides, by setting 20 results per page, and clicking on the link to "page 10", you will be already greeted with the horrendous fishertroll thing🤮

Remember, the false-god was never intended to be a replacement for the original Altavista,
which actually indexed AND showed everything it could index and show...
this is the reason why Gluegow always says
"in order to show you the most IRrelevant results,
blah-blah-blah".

https://forum.videohelp.com/threads/359630-%5BS-E-0-%5D-x265-sucks-P

[StainlessS]

OK, OK, lets see what it says about number of results BEFORE paring them down. [ie the About 29,200 results thingy]

[VoodooFX]

Remember, the false-god was never intended to be a replacement for the original Altavista,
which actually indexed AND showed everything it could index and show...
But everyone switched from Altavista to Google because it suuucked.
Anyway, my Member Berries goes to Astalavista.

[FranceBB]

Whats it gonna say tomorrow.

It's far more than I predicted, with a sharp increase from 29,200 to 37,800 results now! :D

Over to you, StainlessS, for tomorrow's update xD

[Balling]

All works now! https://www.google.com/search?q=https://trac.mpc-hc.org/ticket/5088&filter=0&biw=867&bih=1210&dpr=1.71

[FranceBB]

a sharp increase from 29,200 to 37,800 results now! :D

And we're now at 41,400 results.
Let's keep going, guys! :)

[FranceBB]

OH NO!
We're losing indexing again!

http://forum.doom9.net/robots.txt

is blocking User-agent: Googlebot again!
In fact, we're falling AGAIN!

We just went from 41,400 results to 32,200 results!! :(
Who put the Googlebot block in robots.txt again and most importantly why?!

[nevcairiel]

The robots file does not seem to exist.

[FranceBB]

Yep, they have now deleted it.
Interesting.
I think it's the right choice, though.
Getting rid of it entirely, I mean.
If it's not there, you can't get it wrong hahahahahaha

Anyway, hopefully now all spiders will be able to find us once and for all.

[FranceBB]

Well, deleting the robots.txt seems to be working.
We're bouncing back again:

34,400 results

[StainlessS]

It certainly allowed me to find thread with "guitar" clip back in 2009.

[Balling]

Why does it have both http and https results? You can apparently do 301 server redirect: https://support.google.com/webmasters/thread/107589179?hl=en&msgid=107978668

[filler56789]

Why does it have both http and https results? You can apparently do 301 server redirect: https://support.google.com/webmasters/thread/107589179?hl=en&msgid=107978668

Maybe the admins should have disabled http PLUS the "printable archive" ages ago...
Anyway, it's probable that Google is simply "recycling"
(and recycling very-poorly, to say the least...)
its jurassic databases.
I serious doubt their ""spider"" is actually re-indexing Doom9 from ground-zero.
*woman shrugging emoticon*

Well, deleting the robots.txt seems to be working.
We're bouncing back again:

34,400 results

IMNSHO, the "bigger problem" of Doom9's forum is not it not being indexed well by Gluegow,
the "bigger problem" is: there exists no read-only FULL backup of all its threads and posts on the Web.
Nope, the Wayback Machine is not an option, that site sucks more and more each day, and for various reasons it stops archiving images, ZIP files (attachments), etc, which makes it not-so-useful.
This problem was briefly discussed in the deceased doom10 of Dark Shikari, but nothing practical happened since then and so far :-|

[FranceBB]

IMNSHO, the "bigger problem" of Doom9's forum is that [...] there exists no read-only FULL backup of all its threads and posts on the Web.

Yep, I totally agree.
Although I hope Doom9 isn't gonna go anywhere anytime soon, I do think that we should really have a safe backup of all its contents if, one day, it was to go down for good.
There are too many valuable information and people spent far too many hours of their lives for years and years on it. I mean, it would be a shame if one day it just... disappeared into nothingness.

[FranceBB]

It's been a little over a month now, so I think that it was a good time to see where we're now.
Currently, Doom9 has a total of 153'952 topics and a little over 67'000 have been indexed.
This roughly accounts for 43.5% of the forum.
I can see lots of spiders scanning pages constantly including the Google ones, so it looks like it's just gonna be a matter of time before we get more topics back.
Unfortunately I didn't save those statistics before this whole thing happened, so I don't know where we were before in terms of indexing (I mean in 2022), but I think next month is gonna be a good time to get back and assess the situation again.

See you guys in one month. :)

[Katie Boundary]

Solution: stop using Skynet products and services, and switch to the good guy team: Startpage, Qwant, Duckduckgo.

[StainlessS]

Solution:
Not a solution for the many that do use google, and that is a problem for this site and not for you yourself (or anyone else) as an individual.

[FranceBB]

Speaking of which, two or three days ago Doom9 suffered another outage for a few hours which I believe led to an IP address change, so Google stopped indexing again for a bit.
I thought Swede was gonna add another line to his logs (https://www.leffster.se/) but this time he didn't.
Speaking of which, it might be time to demand a new picture of the closet ehm I mean server room :P

[FranceBB]

It's been a little over a month now, so I think that it was a good time to see where we're now.
Currently, Doom9 has a total of 153'952 topics and a little over 67'000 have been indexed.
This roughly accounts for 43.5% of the forum.

I've been waiting a few more months and we're slowly but surely getting back to full index (although the sporadic downtimes don't make the process any easier).

Anyway, out of 153'952 we now have 79'200 indexed topics, so around 51.44% of the forum.

[FranceBB]

Hey everyone,
ever since the "incident", I keep looking at robots.txt https://forum.doom9.org/robots.txt
Looks like the new robots.txt is now blocking the entire website to the following spyders:

Byte (Chinese Search Engine), Ahrefs (a SEO company), Baidu (Chinese Search Engine) and Yandex (Russian Search Engine)

while Bing and Google are simply delayed so that they don't use too many resources while scanning the forum.
I use Google, so I don't personally care about Chinese and Russian search engines being blocked, but I'm just posting it here in case anyone does.
Still, I think we shouldn't block anything and rather use crawl-delay like we're doing on Google and Bing.

Speaking of Google, we're now at 83,900 pages there, up from 79'200 from 2023.

[Swede]

That's correct. I've been monitoring the load the crawler puts on the server, which is quite heavy, and since Byte and Yandex doesn't honour the crawl-delay as they should (maybe no all but certainly a lot of them) I've just blocked them totally.

[FranceBB]

Ah, gotcha, makes sense. ;)
Thanks as always for the clarification Swede.

[Swede]

Finally put down some hours, found the worst culprits and IPbanned them. I've then reinstated all crawlers and quadrupled the RAM. This should give me some time to relax.