New Googlebot spidering architecture


Most webmasters are familiar with Googlebot’s continual spidering of a web site. Sometimes googlebot traffic can be greater than user traffic, making some web sites like Slashdot to ban googlebot from their servers.

Google has recently addressed the problem. In a move aimed at reducing bandwidth consumption, the adsense spider is now also refreshing the main googlebot cache. The change is shown in the diagram below.

One consequence is that including adsense on your site might get your pages indexed quicker since the Adsense robot is extremely quick.

Matt Cutts, who is provides politic feedback to many a frustrated webmaster and marketing proffessional, confirms that the AdSense Bot is Helping Googlebot With Indexing

Matt warns that you that the Google algorithm will probably penalize those that serve different content to the MediaBot and Googlebot

Leave a Comment

Your email address will not be published. Required fields are marked *