Googlebot Follows Twitter Link Within 5 Seconds. Yahoo! Slurp Within 4.
Something weird has been happening in relation to my Twitter posts.
Always, within a minute, the links have at least 10 hits. Sometimes as many as 20 within 5 minutes.
It happens no matter what time of day I post the tweet.
At this point, I only have 350 followers. And I'm certain they are not poised at their keyboard, finger on mouse, just waiting to pounce on my tweets and follow the links. At least, I hope they're not. I don't post that often.
Being the programmer that I am, I decided to see who was clicking on my links with such alacrity.
I logged the clicks. The link in the tweet was a temporary link meant only for this test. It did redirect to the real page with the real information.
Below is a copy the log for the first 8 minutes after the following tweet was posted:
New: Personalized Animated Cell Phone Screensavers
1283263908 Tue 31 Aug 2010 09:11:48 38.113.234.180 Voyager/1.0 1283263908 Tue 31 Aug 2010 09:11:48 128.242.241.134 Twitterbot/0.1 1283263912 Tue 31 Aug 2010 09:11:52 72.30.142.248 Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) 1283263912 Tue 31 Aug 2010 09:11:52 65.52.2.10 Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0) 1283263912 Tue 31 Aug 2010 09:11:52 67.195.112.234 Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) 1283263913 Tue 31 Aug 2010 09:11:53 66.249.71.237 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) 1283263914 Tue 31 Aug 2010 09:11:54 72.30.142.248 Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) 1283263927 Tue 31 Aug 2010 09:12:07 74.112.128.61 Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8 1283263942 Tue 31 Aug 2010 09:12:22 79.99.6.106 Twingly Recon 1283263943 Tue 31 Aug 2010 09:12:23 204.236.206.79 PostRank/2.0 (postrank.com) 1283264044 Tue 31 Aug 2010 09:14:04 216.24.142.47 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 OneRiot/1.0 (http://www.oneriot.com) 1283264054 Tue 31 Aug 2010 09:14:14 216.24.142.45 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 OneRiot/1.0 (http://www.oneriot.com) 1283264056 Tue 31 Aug 2010 09:14:16 216.24.142.45 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 OneRiot/1.0 (http://www.oneriot.com) 1283264332 Tue 31 Aug 2010 09:18:52 89.151.116.60 Mozilla/5.0 (compatible; MSIE 6.0b; Windows NT 5.0) Gecko/2009011913 Firefox/3.0.6 TweetmemeBot 1283264336 Tue 31 Aug 2010 09:18:56 64.13.147.189 Mozilla/5.0 (compatible; abby/1.0; +http://www.ellerdale.com/crawler.html)
The time 09:11:48 is pretty close to the instant I submitted the tweet. The server clock may be slightly different than my office computer clock. For the purposes of this article, I'll consider the tweet was posted at 09:11:48, the second the first two robots hit the link.
Within 4 seconds after the tweet was posted, Yahoo! Slurp hit the link. A second behind Yahoo! Slurp was Googlebot.
15 hits within the first 8 minutes. 14 of those hits are clearly robots. 1 hit appears to have been a human.
Conclusion: When counting clicks in tweet links, subtract 15 or so from the overall total to get a realistic human clicks count.
The number 15 is a guess and is likely to vary from tweet to tweet. A real number could be obtained by identifying and removing all log entries made by spiders.
Will Bontrager
