Daily Archives: April 20, 2006

New Googlebot spidering architecture

Most webmasters are familiar with Googlebot’s continual spidering of a web site. Sometimes googlebot traffic can be greater than user traffic, making some web sites like Slashdot to ban googlebot from their servers. Google has recently addressed the problem. In a move aimed at reducing bandwidth consumption, the adsense spider is now also refreshing the […]