Re: bad bot behavior

On 3/23/25 9:30 AM, candycanearter07 wrote:

Ian <${send-direct-email-to-news1021-at-jusme-dot-com-if-you-must}@jusme.com> wrote at 12:06 this Wednesday (GMT):

On 2025-03-18, Toaster <toaster@dne3.net> wrote:

>
But what can be done to mitigate this issue? Crawlers and bots ruin the
internet.

>
#mode=evil
>
How about a script that spews out an endless stream of junk from
/usr/share/dict/words, parked on a random URL that's listed in
robots.txt as forbidden. Any bot choosing to chew on that gets what
it deserves, though you might need to bandwidth limit it.

  I heard Cloudflare is doing something like that, but ironically
with their own generative AI..

https://tech.slashdot.org/story/25/03/26/016244/open-source-devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries
They're abusing everyone. If you have a web site, don't allow it to be abused this way.