Friday, September 21, 2012

Are crawlers creepy?

Watching my traffic, it's interesting to see that Google's "crawler" -- the code that goes out across the web seeking to catalog everything that's out there -- visited my most recent post a minute and forty-one seconds after it was posted. This should come as no surprise, since Blogger/Blogspot is owned by Google. It makes sense for Sergei and Larry to have arranged a script to catalog every post that comes out as quickly as it does so, particularly as posting Google adwords on blogs is a revenue stream for them. I dunno how big these dollars are.  Probably small for Google in toto, but big in the incentive comp of the Adwords team, who may well have cut a deal with the crawler team to get such quick service.

But I don't think it's creepy.  I'll take all the traffic I can get for free.

No comments: