Sujet : Re: "Red" And The DoD Language Competition
De : ldo (at) *nospam* nz.invalid (Lawrence D'Oliveiro)
Groupes : comp.lang.adaDate : 12. Sep 2024, 23:27:54
Autres entêtes
Organisation : A noiseless patient Spider
Message-ID : <vbvpta$esm6$8@dont-email.me>
References : 1 2 3 4
User-Agent : Pan/0.160 (Toresk; )
On Wed, 11 Sep 2024 23:57:33 -0500, Randy Brukardt wrote:
"Lawrence D'Oliveiro" <ldo@nz.invalid> wrote in message
news:vbimad$1j26j$6@dont-email.me...
>
Don't you wonder why people insist on returning "403
Forbidden" for those using a command-line tool like wget?
Most SEO tools and other
useless/criminal scrapers like to fake their identification, and WGet is
a favorite for that task. There are many owners that block that and many
other abused user-agents.
That doesn’t make any sense, because anybody who knows how to use wget
would know about its “--user-agent” option. So if they really were using
wget to conduct their site abuse, you wouldn’t know, and blocking wget’s
default user-agent setting wouldn’t help.
(The Ada-Auth.org blocks about 20 user-agents, but not WGet.
Which kind of proves my point.
“Red” And The DoD Language Competition
Re: “Red” And The DoD Language Competition