At Home with the Robots: 2017 edition

Further Afield

Not necessarily search engines, but still legitimate in my book:

Qwantify
IP: 194.187.170-171
UA:

Mozilla/5.0 (compatible; Qwantify/2.3w; +https://www.qwant.com/)/2.3w

Mozilla/5.0 (compatible; Qwantify/2.4w; +https://www.qwant.com/)/2.4w

Qwantify/1.0

The third, minimalist UA is only for the favicon. The change from 2.3 to 2.4 happened around the 24th, with no overlap.

MJ12bot
Is it a search engine? Is it something else? You decide. Either way, MJ12 is one of the best-known distributed crawlers, so I won’t bother to list its IPs.
UA:

Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/)

Qwantify and MJ12 are among the few law-abiding robots that still use HTTP/1.0.

Cliqzbot
UA:

Mozilla/5.0 (compatible; Cliqzbot/1.0; +http://cliqz.com/company/cliqzbot)

“Was genau ist Cliqzbot?” Another of those targeted searches, I think. They are either distributed, or they sprawl so widely across 52 that there’s no telling where they really live.

DeuSu
IP: 85.93.91.84
UA:

Mozilla/5.0 (compatible; DeuSu/5.0.2; +https://deusu.de/robot.html)

DeuSu only understands Disallow if they’re given a section to themselves in robots.txt.

Yeti
IP: 125.209
UA:

Mozilla/5.0 (compatible; Yeti/1.1; +http://naver.me/bot)

Although they only showed up once--robots.txt plus a page--Yeti deserves a look-in because this month marks the first sighting since ... drumroll ... July of 2014. That was at my old site, which they used to visit all the time; they’ve never before set foot on my main site. They used to change IP every year or so, but this one's been the same since mid-2013.

Three Headscratchers

Bing and Yandex don’t have any connection that I know of--but they’re both associated with one near-identical behavior:

Visitor comes in with generally humanoid headers and requests a page, sometimes giving the appropriate search engine as referer. They request scripts, stylesheets, fonts--in short, all supporting files except images and favicon. All three entities pick up piwik.js, though only one of them acts on it by requesting piwik.php. (It may or may not be relevant that piwik lives on a different site, although presumably on the same server.)

Drake Holdings:
IP: 204.79.180 (the range belongs to an outfit called Drake Holdings, hence the name)
UA:

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/5.0;  Trident/5.0)

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0;  Trident/5.0)

That’s two  spaces before the second “Trident”. Any given visit uses one UA or the other, but apart from that they’re random. Some html requests have no referer; some have bing search, with a plausible search string for the requested page. Not always right, but always understandable. Requests for supporting files have human-type referers. Unlike the other two members of the trio, it requests piwik.php, meaning that it acts on javascript.

Plainclothes Bingbot:
IP: 65.55 and 131.253
UA:

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0;  Trident/5.0)

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/5.0;  Trident/5.0)

(i.e. identical to the two Drake Holdings forms)

Both of these have been getting 302 redirected to a custom page that has served different purposes over the years; one of its functions is to intercept humans who accidentally behave like robots.

Yandex Referers
IP and UA: as expected for humans in Russia
Referer: http://yandex.ru/clck/jsredir?from=yandex.ru%3Bsearch%3Bweb%3B%3B&text=&etext= ... et cetera, et cetera, with an enormously long string of garbage that appears to be identical to a genuine Yandex referer. Each request is followed or preceded within 24 hours by an apparently human visit (with images and piwik, without favicon) to the same page. Different IP and UA, but Yandex referers to my site are infrequent enough that I can easily pick them out.

Bingbot’s Evil Twin
Alongside the ordinary plainclothes Bingbot, there’s this further agent that I never noticed until fine-tooth-combing logs:

IP: 40.77.169
UA:

Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36

I had no idea this Chrome UA existed; they’ve been getting a 403 on the grounds of “non-bingbot from Bing/MSN range”. They make the same set of three requests every few days: first /dir/subdir1/ and then, several hours later, /dir/subdir2 (without slash) immediately redirected to /dir/subdir2/ with slash. (Paradoxically, the malformed URL is what prevents it from being blocked at the outset; it’s a very narrowly constrained RewriteRule.) Turns out this has been going on--always the same set of three--since mid-September.

:: insert “wtf?” emoticon here ::

Other Law-Abiding Robots

In alphabetical order:

AhrefsBot
UA:

Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)

Probably distributed; I counted eight different ranges this month. But all robots.txt requests come from just two IPs, 5.196.87.7 and 151.80.31.151, which may explain why their website says that robots.txt changes can take up to a week to be recognized. In spite of this, they’ve never asked for anything in a roboted-out directory. Requests are mostly pages, with a few seemingly random images mixed in.

Applebot
IP: 17.142
UA:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)

Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B410 Safari/600.1.4 (Applebot/0.1; +http://www.apple.com/go/applebot)

The iPhone version is rare. Has anyone ever figured out what this robot does? Some people may remember “If robots instructions don’t mention Applebot but do mention Googlebot, the Apple robot will follow Googlebot instructions.” The Applebot is not the only robot to adhere to this quaint misapprehension.

BLEXBot
IP: 148.251.244.204
UA:

Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)

“BLEXBot assists internet marketers to get information on the link structure”

I am not absolutely certain it is wise to put the word “BLEXBot” and “link structure” into the same sentence. Surprisingly, only about 1/6 of the month’s requests were 404s caused by appending other people’s URLs to my paths--a pervasive problem that others have noticed too. I would have guessed closer to 95%. It looks as if they’ve had the problem since December 2016, but it has been getting worse.

BUbiNG (open source)
IP: 64.62.252, 104.254.132.10
UA:

BUbiNG (+http://law.di.unimi.it/BUbiNG.html)

BUbiNG is a scalable, fully distributed crawler, currently under development and that supersedes UbiCrawler.

Although the two IP ranges belong to different hosts, there are no major differences in their behavior. (UbiCrawler must have been before my time; I find no record of it.) I put them in the “No skin off my nose” category.

DotBot
IP: 216.244.66.229
UA:

Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)

Psst! DotBot! It’s tidier when the URL in your UA string doesn’t redirect--especially not to an entirely different domain. (They are not the only ones.)

SemrushBot
IP: mostly 46.229.168.75 (the exceptions may be fakers, but why would you pretend to be the SemrushBot?)
UA:

Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)

It would be very interesting to know what they’re looking for, since several requests were for obscure interior non-English-language pages that, to the best of my knowledge, are not linked from anywhere in the known universe.

SEOkicks
IP: 138.201.30.66
UA:

Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)

SiteExplorer
IP: 208.43.225.84-85
UA:

Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)

SiteExplorer is one of the rare robots that only understand Disallow if they’re given a sector to themselves in robots.txt, so I initially thought they were non-compliant. Later evidence suggests that they’re just very, very slow on the uptake: in the course of the month they picked up robots.txt eleven times, but never screwed up the courage to ask for a page until almost the end of the month.

spbot
IP: 138.197.74.117, 45.55.175.162 (two full crawls)
UA:

Mozilla/5.0 (compatible; spbot/5.0.3; +http://OpenLinkProfiler.org/bot )

I think their crawling happens on the fly: robots.txt, two forms of root--one of which gets a 301--and then all other pages, from top to bottom, with the same referer a human would send. In the rare case that a page is linked from widely separated directories on the same site, the referer is whichever one the robot saw first. Since they don’t come in with a shopping list, there are never any 301s or 410s. This makes it useful for record-keeping purposes: Count the number of requests, subtract two, and that’s how many visible URLs you’ve got ;)

test Crawl
IP: 54.92.230.108 (always)
UA:

test Crawl

I did say I didn’t have very high standards when it comes to authorizing robots. They were very active in the latter months of 2016; in April they only showed up once. They’re only interested in one directory: mostly pages, but the occasional stylesheet, and sometimes the first image on a page--regardless of whether it’s a full-color frontispiece or a little icon from the navigation banner.

UptimeBot
UA:

Mozilla/5.0 (compatible; Uptimebot/1.0; +http://www.uptime.com/uptimebot)

Referer:

http://uptime.com/example.com

Like SiteExplorer, UptimeBot is exceedingly slow w/r/t to robots.txt, but if you keep denying them for a month or two they will eventually get the message and stop requesting pages. At that point you may choose to whitelist them. Most of the time they just do a HEAD / which is hardly a server-intensive request.

CCBot
IP: 54.162.105.83, 54.225.9.188 (they pick a fresh IP for each crawl, and don’t come around very often)
UA:

CCBot/2.0 (http://commoncrawl.org/faq/)

May be following outside links, though they did once look at the sitemap. Their web page says they’re Nutch-based, which may explain their compliance with robots.txt--not something you see every day from the 54 neighborhood.

Targeted Robots: Authorized

Most of these first showed themselves in the latter part of 2015 when I started putting ebooks into a curated directory that includes both an RSS feed and a “New Listings” page. These robots tend to be truthful about where they learned about the page they’re requesting, even if they didn’t literally click on a link to get there, the way a human would.

Flipboard
UA:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:49.0) Gecko/20100101 Firefox/49.0 (FlipboardProxy/1.2; +http://flipboard.com/browserproxy)

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.6; +http://flipboard.com/browserproxy)

Flipboard 1.2 is for robots.txt and pages, 1.6 is for images. For each new file, they request the HTML a few times, and the associated images just once. They don’t seem to be interested in stylesheets.

omgili
IP: 82.166.195.65
UA:

omgili/0.5 +http://omgili.com

The name is short for OMG I Love It. Unfortunately, I am not making this up. Check for yourself. This is a recent arrival; I only started seeing it earlier this year. It picks up a page when it first learns about it, and then comes back every week or so for the same page.

Less frequent visitors include:

magpie-crawler: 185.25.32.42
trendictionbot: 144.76.22.175
rogerbot (Nutch-based robot from Mozilla)

Social Links

facebook
IP: 31.13.64-127 (Ireland), 66.220.144-159, 69.63.128-191, 69.171.224-255, 173.252.64-127
UA:

facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

facebookexternalhit/1.1

I have not yet figured out the significance of the shorter UA. It made its first appearance in the latter part of 2015, but doesn’t seem to be a direct replacement of any other UA, and doesn’t have a clearly recognizable function. The formerly common visionutils has not been around since April 2016.

Facebook’s behavior changes every couple of years. Currently new links start with a page request, followed by all image files belonging to the page, giving the page as referer. The human user then presumably selects one image, which will get re-requested every time your original visitor’s friends view the page that linked to you. These image-only requests come in with no referer. (Pro tip: Do your pages include a non-visible image file, such as piwik’s noscript dot? I rewrite this for facebook to a little banner giving the site name. So if a page happens not to have any good pictures--some books don’t--there’s still something for the human user to select.)

Facebook, incidentally, seems to be responsible for those /dir/subdir2 without-final-slash hits, now limited to the Bingbot Evil Twin. All it takes is one person to like a page and misspell its URL...

twitter
IP: 199.16.156-159, 199.59.148-151
UA:

Twitterbot/1.0

Uncharacteristically for a social-media-based robot, the Twitterbot asks for and obeys robots.txt. And, like the Googlebot, it never forgets. This month’s requests included an URL that I remember seeing on Redirect lists, meaning that they first learned about it no later than December 2013.

NetVibes
IP: 193.189.143
UA:

Netvibes (http://www.netvibes.com)

Fakers
In the category of Most Unlikely User-Agent:

UA: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0

The IP varies, but it’s always the identical UA string from beginning to end, so it’s not just an extra bit tacked on to the end of a human browser.

Leftovers

Finally, on the Probably-Good side, anyone who shows pretensions to robotitude by asking for--and honoring--robots.txt and having an identifiable name:

crawler4j
DomainCrawler
GarlikCrawler
MojeekBot
SafeDNSBot
SemanticScholarBot
SurveyBot (DomainTools)
VeriCiteCrawler
Wotbox

The name is crucial; there’s not much point to asking for robots.txt if you’re going around calling yourself Firefox 10, or even Chrome 54.

To be continued...

And now the bad news...

Well, not that bad. Overall, I think about 3% of unwanted robots got in--and most of those just picked up the front page and went on their way. I didn’t see any really devastating crawls coming out of nowhere. It isn’t enough to have a thoroughly human UA; they also have to send plausible humanoid headers. Thankfully, most robots are either too stupid or too lazy. About 3% of them sent no UA at all: instant lockout. About one-quarter--including a number of major search engines--did not send an Accept header: instant lockout unless whitelisted. (Other header anomalies are, of course, for me to know and you to find out. Or not.)

Robotic User-Agents

Aside from that small group that can’t be bothered to send a UA header at all, there are always a few brand-new crawlers who go around for months with the equivalent of Insert Name Here. And then there are the computer-science class assignments who never do figure out what to call themselves. (“Am I ‘Test1’? No, wait, I think I was ‘testone’.”) The fake Googlebot seems to have all but disappeared in recent months. Maybe the people who write robot scripts have figured out that “Googlebot UA + non-Google IP = automatic lockout”, so it ends up being worse than useless.

This month, about 85% of all robots were smart enough to start their names with “Mozilla” followed by some more-or-less-plausible humanoid user-agent. Firefox/40.1 seems to be in fashion just now; at least it’s a little more believable than the assorted one-digit Firefoxes. And a visitor of any species, human or robot, calling itself MSIE 6 ... can only inspire pity.

Some of them, though, aren’t even trying:

null
IP: 84.109.137.242
UA:

null

Go-http
UA:

Go-http-client/1.1

The pattern of requests suggests that most of them are the same robot, in spite of coming from all over the map.

Dorado WAP-Browser
UA:

Dorado WAP-Browser/1.0.0

Sometimes humans wear this face too, but more often it’s a robot.

Not Welcome Here

On some sites, Chinese search engines might be perfectly legitimate and even desirable. Me, I want no part of ’em--but try getting this message through.

Baidu
IP: 123.125.71, 180.76
UA:

Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)

Mozilla/5.0 (Windows NT 5.1; rv:6.0.2) Gecko/20100101 Firefox/6.0.2

In April 2017, the Baiduspider by that name never showed its face at all. (It isn’t gone; I’ve seen it in May.) It visited once or twice a day from 180.76, but only to request robots.txt under the FF 6 alias. There were also a slew of blocked root requests throughout the month, from 123.125.67--an IP that Baidu has used in the past--claiming to be Chrome/45.

What I did see was a robot from an ARIN range professing to be

compatible;Baiduspider/2.0; +http://www.baidu.com/search/spider.html

[sic] What’s better than faking your UA? Claiming to be something that would be banned in its own right.

Sogou
IP: 36.110, 106.38, 106.120, 123.126, 220.181.124
UA:

Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)

Currently it doesn’t seem to be interested in much but /ebooks/, which strongly suggests it is picking up links from some outside source. Requests but ignores robots.txt, where it is denied by name.

In both cases--Baidu and Sogou--I don’t know whether the robot is wilfully ignoring robots.txt, or whether it’s too stupid to understand a continuous listing (more than one User-Agent line sharing a single Disallow directive).

The Worst of the Worst

When you walk in and demand to see /wp-login.php, it’s all over. There is zero chance that you have any legitimate, law-abiding purpose. (I suppose if you actually have a WP site, you might invite security-testing robots to look around and confirm that nobody can get in where they don’t belong. That is not what my visitors are up to.) Close to 10% of the month’s robotic visits asked for wp-login, wp-admin, Fckeditor, xmlrpc.php and similar. (What they received, instead, was the 403 page--or, at worst, a manual 404.) A surprising number asked for robots.txt, or the meaningless “/blog/robots.txt”. I’m not clear about the purpose of the request: they did not rush over and ask for any and all roboted-out directories, but they did go ahead and ask for the standard wp list. Are they hoping to find something like “Disallow: /wp-admin876” that isn’t worth asking for unless you already know it exists?

The main feature of malign robots is miscellaneousness. Some IP shows up, requests two or six or a dozen files from a standard list, and never shows its face again. Repeat visits are a feature of named robots from known addresses.

Interestingly, about about half of the group--from every possible IP--came in with the exact UA

Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1

suggesting that they all started with the same script. Unfortunately, the identical UA is still in use by humans. But many others had no UA, making for a convenient Shoot To Kill twofer.

Request Patterns

Most malign robots can be identified by their behavior rather than by name and address. Current patterns:

1+3
GET /some-inner-page (always a different one) giving root as referer although the requested page is not actually linked from the root
GET / (root), giving the previously failed request as referer
GET / (root), with auto-referer
GET / (root), with auto-referer
This month--Hurrah!--every last one of this pattern was blocked.

Contact Botnet
GET /some-inner-page (always a different one) giving root as referer, as above
GET / (root) with auto-referer
GET /boilerplate/contact.html giving root as referer
GET / (root) with auto-referer
These requests are always blocked--but only thanks to headers. IP alone won’t do it; the “Contact botnet” plagued me for years. A further quirk of this pattern is that the first referer has a final / slash, but the other three don’t. And, of course, they must have friends inside, because the first request is for some page that exists but is not visible from the front page.

Multiples:
Only one page is requested--but they ask for it repeatedly, anywhere from 3 to 7 times, most often 4. Sometimes they throw in some referer spam. Almost all of them are blocked.

One is enough:
IP: various
UA and headers: sufficiently humanoid
Request: one page from one directory ... which happens to consist of unusually large HTML files. I would prefer not to send out 300K if I don’t have to, so now they get 302 redirected to a detour page at about 1/200 the weight. (The page also catches the rare human by mistake, but they’re given enough information to proceed normally.) To make these go away I’ve been reduced to blocking a handful of IP ranges--which is exactly what I’d hoped to get away from by changing to header-based access controls. Sometimes there is just no other way, darn it.

“Your root page sent me”
Referer: example.com (with or without final /)
Claiming that you got to some deep inner page directly from the root is a good way to get yourself blocked--especially if also you get the www wrong. Sending an auto-referer for the root itself is similarly effective. Most of them don’t even get as far as the RewriteRules, though.

The Current Faker

Calling yourself Googlebot seems to be going out of fashion. The one I saw most often didn’t even use the full UA string; it thought it could get away with the magic word alone:

Googlebot (gocrawl v0.4)

Even then, it hedged its bets; the “gocrawl” version alternated with

Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120716 Firefox/15.0a2

which counts as “could be better, could be worse” among humanoid UAs. All blocked, so no skin off my nose in any case.

Targeted Robots: Unauthorized

I talked earlier about robots that read an RSS feed, or a certain site’s New Releases page, and come in asking for a file from that list. The Great Divide is whether they first ask for robots.txt. These don’t:

PaperLiBot
IP: 37.187.165.31
UA:

Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)

NewsBlur
UA:

NewsBlur Content Fetcher - 61 subscribers - http://www.newsblur.com/site/18645/new-online-books (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)

Yes, it really says “61 subscribers” right there in the UA string. At least this month.

metadataparser
UA:

metadataparser/1.1.0 (https://github.com/bloglovin/metadataparser)

AppEngine
IP: 107.178.194
UA:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36 AppEngine-Google; (+http://code.google.com/appengine; appid: s~feedly-nikon3)

In the past they have had other, similar UAs, but for now they seem to have settled on this form. Each time they are presented with a new title, they request it over and over for a week or so and then lose interest.

Thanks but No Thanks

On second thought, it isn’t enough to ask for robots.txt. You also have to do what it says. That means:

MegaIndex
IP: 176.9.50.244 (formerly at 88.198.48.46--they seem to have taken 2016 off, and returned at a new address)
UA:

Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)

Some robots request robots.txt only after getting the front page; this is one of them. But, since they proceeded to ask for the entire contents of a roboted-out directory, it becomes pretty academic.

James BOT
UA:

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 - James BOT - WebCrawler http://cognitiveseo.com/bot.html

linkdex
IP: various 54
UA:

Mozilla/5.0 (compatible; linkdexbot/2.2; +http://www.linkdex.com/bots/)

ltx71
IP: 52.2.10.94 and 104.154.58.95 (exactly)
UA:

ltx71 - (http://ltx71.com/)

I first became aware of this robot when it showed up requesting new ebooks, but it turns out it also likes the page linked from my profile here; I just never noticed because it was always blocked for one reason or another.

NetShelter
UA:

Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0 (NetShelter ContentScan, contact abuse@inpwrd.com for information)

WBSearchBot
UA:

Mozilla/5.0 (compatible; WBSearchBot/1.1; +http://www.warebay.com/bot.html)

We don’t want your kind around here

If they never even ask for robots.txt, how will they know what it says?

Gluten Free Crawler
IP: 104.131.147.112
UA:

Mozilla/5.0 (compatible; Gluten Free Crawler/1.0; +http://glutenfreepleasure.com/)

Crawls URLs it finds listed on other sites--including but not limited to the one given in my WebmasterWorld profile. As far as I can tell, the name is meant as a joke, not as referer spam. Ancient history: Someone from this exact IP, though using a different name, asked for robots.txt on 24 November 2014. Apparently they’re still assimilating its contents; they’ve never asked for a fresh copy.

Inbound Links
IP: 192.157.254.197 and 199.193.251.133 in alternation
UA:

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)

Request: HEAD /
Referer:

https://www.inboundlinks.win/InboundLinks/www.example.com 

(always with incorrect www) They’re persistent, I’ll give them that; every few days they’re rattling the doorknob again.

Others

djbot/1.1 (+http://www.demandjump.com/company/about)

OrgProbe/0.9.4 (+http://www.blocked.org.uk)

Xenu Link Sleuth/1.3.8

I don’t know and don’t especially care whether this is the actual Xenu; all I know is, I didn’t order it. (I don’t know about Xenu’s ordinary behavior. The w3 link checker requests robots.txt on each site that it visits, and goes away weeping if it doesn’t find authorization.)

This Month’s Winner ...

... in the “Extra Stupid” category is:

User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31

And your point is...? Only that the element “User-Agent:” is part of the UA string. I also met a lone

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:x.x.x) Gecko/20041107 Firefox/x.x

which similarly failed to read the Your New Robot instructions carefully enough.

“That’s All Folks!”

[edited by: lucy24 at 4:16 am (utc) on May 13, 2017]

Great comprehensive post Lucy24!

in the “Extra Stupid” category is:

User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31

And your point is...? Only that the element “User-Agent:” is part of the UA string

Many test drive versions (github) offer the user the ability to customize the UA field. A couple I've seen identify that field as "User-Agent:" but it remains in the UA string unless the user actually customizes that header field.

So this is actually an easy way to spot a fake browser and block. Too bad they all don't do it that way.

This has got to be the most exhaustive "State of the Robots" resource anywhere. Thank you lucy24 for keeping notes and compiling the data and sharing it here! Not just this year, but the history, habits and evolution of these bots is so useful when some puzzling behavior pops up. Much appreciated! :)

Raw logs have the data, but it takes a human to mine the data to come up with actionable results. Kudos! And thanks!

Indeed, a great walk through write up! Thanks much.

I love this forum because (1) most posters (waves to lucy and keyplr) are more open than I and share their logic, which is a fabulous check on mine own; (2) there are always edge cases and new hazards to beware that simply are not shared elsewhere.

This forum is probably WebmasterWorld's greatest unappreciated treasure.

Unfortunately, the folks who really could benefit from this.... Ah, well.

Thank you lucy24 for posting this comprehensive information.

Without doubt, it's a document to be bookmarked as it's extremely valuable.

It's worth noting that this report is specific to the author's experience. Others may see a different set of User Agents (robots.)

Also, what is unwanted at one website may be valued at another. Each webmaster must determine whether each agent is something beneficial, to be tolerated, or unwanted.

True, but most of the time lucy24 nails it as to the unwanted. I'll start there an then make my own decisions on what to keep or disallow. :)

Both Baidu and Sogou are well behaved (YMMV) bots from huge Asian Search Engines and as such are a high value resource even if you do not wish traffic from these regions. The problem is, the UAs are faked often and rDNS is tricky because of the parent ISPs.

Into each life a little rain must fall, and in the process one weeds out the unwanted and makes welcome those of value. Oddly enough (info site) Asian, EU, and RU are very active... so I spend time dealing with the bad actors as they show up.

this report is specific to the author's experience

Or, if you like,
This report is specific to the author's experience
;)

Botnets come, botnets go. Even some perfectly well-mannered robots, such as minor search engines, do their crawls at widely spaced intervals (for example I'm currently witnessing Mail.RU's semiannual Catchup Crawl) so one month's data may be widely different from a month before or after.

Ha ha... well I didn't intend it to sound like a dire warning; just trying to keep it academic for readers with various levels of experience.

I Love Lucy . . . . . . . . . . . . . . was a funny TV show in the 1950s.