How to Save Googlebot the Trouble of Crawling Infinite Space!

Aug 6, 2008 | 2,509 views | by Navneet Kaushal
VN:F [1.9.20_1166]
Rating: 5.0/5 (1 vote cast)

Over at the Google Webmaster Central Blog, Google has put up an interesting post about how Googlebot can get stuck in the loop of 'infinite space' and what measures can the Webmasters take to avoid such a glitch.

Infinite Space as defined by Google: “These are very large numbers of links that usually provide little or no new content for Googlebot to index. If this happens on your site, crawling those URLs may use unnecessary bandwidth, and could result in Googlebot failing to completely index the real content on your site.”

However, with the increase in such incidents, Google has now started to inform the Webmasters via the Webmaster Tools message center, whenever such an anomaly is discovered. Because the issue of infinite space can cause inconvenience to the Webmasters, therefore it is imperative that they get their websites verified with Webmaster Tools. Frequent check of the Webmaster Tools message center would also serve as an added advantage.

Google Webmaster Tools Message Center:

google11 How to Save Googlebot the Trouble of Crawling Infinite Space!

Example of an Infinite Space:

Google has taken the example of a calendar with a 'next month' link. As per Google, “It may be possible to keep following those "Next Month" links forever! Of course, that's not what you want Googlebot to do. Googlebot is smart enough to figure out some of those on its own, but there are a lot of ways to create an infinite space and we may not detect all of them.”

event calander How to Save Googlebot the Trouble of Crawling Infinite Space!

How to Avoid/Correct the Infinite Space Issue:

However, while explaining the problem of Infinite Space, Google has also provided some tips as to how Webmasters can avoid and correct this anomaly.

  1. Go through the Webmaster Tools Help article, where you can know more about the steps that can be taken to avoid infinite space.
  2. Eliminate whole categories of dynamically generated links using the robots.txt file. You can learn more about the usage of the robots.txt file for the elimination of infinite space here.
  3. You can also block these useless links with a "nofollow" link attribute. You can learn more about the usage of the “nofollow” link attribute here.
How to Save Googlebot the Trouble of Crawling Infinite Space! , 5.0 out of 5 based on 1 rating
4.thumbnail How to Save Googlebot the Trouble of Crawling Infinite Space!

Navneet Kaushal

Navneet Kaushal is the founder and CEO of PageTraffic, an SEO Agency in India with offices in Chicago, Mumbai and London. A leading search strategist, Navneet helps clients maintain an edge in search engines and the online media. Navneet's expertise has established PageTraffic as one of the most awarded and successful search marketing agencies.
4.thumbnail How to Save Googlebot the Trouble of Crawling Infinite Space!
4.thumbnail How to Save Googlebot the Trouble of Crawling Infinite Space!