Are other instance admins running into AI crawlers impacting availability?
-
A general call-out to #fediadmin about whether they have been experiencing issues with site availability, specifically an on-again off-again bot wave that crawls publicly-available ActivityPub data at a rate so punishing that it overwhelms my server.
We had to turn on “I’m Under Attack” from the CF dashboard, but that comes with some not-so-nice tradeoffs (including blocking all automated crawlers, even the good ones.)
I initially thought someone just didn’t like us (community.nodebb.org, actually), but now we’ve received reports of it happening to another NodeBB, and silverpill@mitra.social’s mitra.social is down for the count currently.
Admittedly, that last one could be unrelated.
-
A general call-out to #fediadmin about whether they have been experiencing issues with site availability, specifically an on-again off-again bot wave that crawls publicly-available ActivityPub data at a rate so punishing that it overwhelms my server.
We had to turn on “I’m Under Attack” from the CF dashboard, but that comes with some not-so-nice tradeoffs (including blocking all automated crawlers, even the good ones.)
I initially thought someone just didn’t like us (community.nodebb.org, actually), but now we’ve received reports of it happening to another NodeBB, and silverpill@mitra.social’s mitra.social is down for the count currently.
Admittedly, that last one could be unrelated.
Yes it’s affecting everyone.
The way you described it, it sounds like they’re making ActivityPub requests, with the
Accept: application/activity+jsonheader set?? I haven’t seen that, they all just make normal web requests.Some instances have set up Anubis but I found it just about impossible to get configured right, in a way that doesn’t break federation or api access.
I found that the user agent strings they use for their requests have random numbers for the browser version so if you craft a nginx regex-based config that blocks really old browsers, that gets rid of 95% of it.
Also in cloudflare you can block entire ASNs in the security rules area. 136907 and 45899 are block-worthy.
Also https://community.nodebb.org/robots.txt is extremely weak. Compare that with https://piefed.social/robots.txt
-
>mitra.social is down
There was an outage at the data center somewhere. I haven't noticed any unusual crawler activity.
-
Yes it’s affecting everyone.
The way you described it, it sounds like they’re making ActivityPub requests, with the
Accept: application/activity+jsonheader set?? I haven’t seen that, they all just make normal web requests.Some instances have set up Anubis but I found it just about impossible to get configured right, in a way that doesn’t break federation or api access.
I found that the user agent strings they use for their requests have random numbers for the browser version so if you craft a nginx regex-based config that blocks really old browsers, that gets rid of 95% of it.
Also in cloudflare you can block entire ASNs in the security rules area. 136907 and 45899 are block-worthy.
Also https://community.nodebb.org/robots.txt is extremely weak. Compare that with https://piefed.social/robots.txt
rimu@piefed.social said in Are other instance admins running into AI crawlers impacting availability?:
> The way you described it, it sounds like they’re making ActivityPub requests, with the Accept: application/activity+json header set?? I haven’t seen that, they all just make normal web requestsThey’re just regular requests. Just hitting routes that aren’t local to the NodeBB (they’re probably getting them from our version of the federated timeline.)
Thanks for the actionable recommendations! A lot of this is easy to do if you use CloudFlare, but would be harder if you don’t have that option available to you.
I always thoughts
robots.txtwas basically pointless. We use it as guidance for SEO purposes, but these crawlers don’t respect it do they??! -
The ones in the PieFed robots.txt no longer appear in my nginx logs. It can take a few days for them to notice but it works eventually. They are generally well behaved (1 request per second, maximum) but collectively they still add up to quite something.
The truly naughty scrapers, like those with the fake user agents, yeah they ignore it.
-
R ActivityRelay shared this topic
-
The ones in the PieFed robots.txt no longer appear in my nginx logs. It can take a few days for them to notice but it works eventually. They are generally well behaved (1 request per second, maximum) but collectively they still add up to quite something.
The truly naughty scrapers, like those with the fake user agents, yeah they ignore it.
See that’s the thing, I don’t know if I want to block all AI crawlers. I’m not bullish on AI at all, and I do think that it’s a huge waste of resources for these companies to proactively crawl everything they can get their hands on, but by and large it’s not really been a major issue.
I could be wrong… I know @baris did do some work on attempting to block some of these bots back when they were being quite egregious too.
The wave we’re seeing are all using common user agents and rotating IP addresses from Brazil/Vietnam/etc.
I’d try a geo-block next I think. Possibly Anubis, as well as this little tidbit you mentioned above:
> I found that the user agent strings they use for their requests have random numbers for the browser version so if you craft a nginx regex-based config that blocks really old browsers, that gets rid of 95% of it.
I need to inspect the user agents more closely next wave…
