May 2, 2008 114 reads by Navneet Kaushal

Numerous Webmasters have reported that Google crawlers also known as GoogleBots have recorded their slowest crawling graph yet. For the month of April, the crawling rate fell by almost 75% almost multiple websites.

In a forum thread at Webmaster World there are numerous reports of the same phenomenon that is giving Webmasters sleepless nights. Some of the excerpts from this thread are mentioned below:

“I monitor several sites with more than ten thousand pages. On all of those sites the GoogleBot crawl rate has dropped by at least 75%. Is this a phenomenon that others are seeing as well.

I am observing this on my own site, and sites run by several others. The sites are not related (different ownership, development teams, link strategies, hosting facilities, etc).

The graphs in Google webmaster tools (site -> tools -> set crawl rate) show that pages and kilobytes per day have dropped drastically, almost to zero in April. The graph of time spent downloading a page has not changed.

Are other people observing this phenomenon too?”

“I can verify that our main site is having the exact same thing occur. The crawling stats were not updated for most of March and when they came back in April, they were almost totally flat.

Our logs verify that googlebot is now 3rd behind Y and then MSN. For the last few years it was always first… by a large margin.”

“I see 90% less crawling and indexing all over the place, maybe they just need their horse power to crank out another penalty?”

“Glad to see this topic…I have millions of pages spread across hundreds of sitemaps. I parse my logs daily to see what Googlebot is doing, and my totals for April are 1) way down from previous months and 2) much more volatile from day to day. I saw a lot of days in February and March in the 150,000-200,000 range of pages crawled per day. By comparison, here are the last couple weeks:

For example:
4/14/2008 139,279
4/15/2008 109,410
4/16/2008 83,809
4/17/2008 2,019
4/18/2008 79,594
4/19/2008 95,466
4/20/2008 92,461
4/21/2008 101,422
4/22/2008 8,410
4/23/2008 3,195
4/24/2008 3,667
4/25/2008 69,842
4/26/2008 65,134
4/27/2008 88,823
4/28/2008 117,792
4/29/2008 123,555
(note that these numbers include both general spidering and sitemap crawling)

Those three days starting 4/22 had me tearing my hair out. I did an early check on today, and I can see the total is going to be considerably down again from yesterday (probably sub-50,000).

A month ago I was hoping to spend some time seeing if I could speed my pages up even more to get Googlebot firing through my site even faster…now I'm afraid to do anything, since spikes in the crawl could be a result of something I screwed up or Googlebot just deciding to shut down for three days.

Back in March I introduced gzip to my site–I expected it to speed up Googlebot, but it seems to have done nothing…but then again, with spikes like these, how would I know?”

Navneet Kaushal

Navneet Kaushal

Navneet Kaushal is the founder and CEO of PageTraffic, an SEO Agency in India with offices in Chicago, Mumbai and London. A leading search strategist, Navneet helps clients maintain an edge in search engines and the online media. Navneet's expertise has established PageTraffic as one of the most awarded and successful search marketing agencies.
Navneet Kaushal
Navneet Kaushal
Most popular Posts
Upcoming Events
Events are coming soon, stay tuned!More