Re: bad bot behavior

On 3/18/25 10:17 AM, Ben Collver wrote:

Please stop externalizing your costs directly into my face
==========================================================
March 17, 2025 on Drew DeVault's blog
 Over the past few months, instead of working on our priorities at
SourceHut, I have spent anywhere from 20-100% of my time in any given
week mitigating hyper-aggressive LLM crawlers at scale.

This is happening at my little web site, and if you have a web site, it's happening to you too. Don't be a victim.
Actually, I've been wondering where they're storing all this data; and how much duplicate data is stored from separate parties all scraping the web simultaneously, but independently.