Re: bad bot behavior

Ian <${send-direct-email-to-news1021-at-jusme-dot-com-if-you-must}@jusme.com> wrote:

On 2025-03-18, Toaster <toaster@dne3.net> wrote:

>
But what can be done to mitigate this issue? Crawlers and bots ruin the
internet.

 
#mode=evil
 
How about a script that spews out an endless stream of junk from
/usr/share/dict/words, parked on a random URL that's listed in
robots.txt as forbidden. Any bot choosing to chew on that gets what
it deserves, though you might need to bandwidth limit it.

Another option could be to craft a "gzip bomb" (a carefully crafted
zlib compressed file that is compressed to the maximum limits of the
zlib/gzip algorithm) and return it with the http type of "gzip
compressed".

Then you only have to output a few tens of megs, but if the AI
decompresses the gzip bomb it has to consume multiple gigabytes of
data.