Aug 18, 2008 116 reads by Navneet Kaushal

At the beginning of this month, we had informed our readers that Google had allegedly blocked one of the most popular search engine ranking softwares, WebPosition.

Now Search Engine Roundtable reports that, over at the Google Groups, Matt Cutts and JohnMu have finally spoken up about why and how Google blocks ranking checking tools.

This is what Matt Cutts had to say:

“Hi Scott. Google does use algorithms and different techniques to block
excessive automated queries and scraping, especially when a someone is
hitting Google quite hard. The reason is that scraping consumes server
resources. We don't want real users to be slowed down or affected just
because a bot is sending bunches of automated queries to Google.

We do turn off a number of tools/bots/IP addresses that scrape us too
heavily. It's a common enough phenomenon that we did a blog post on
Google's Online Security Blog about the subject.
In fact, I know that just a week or so ago our algorithms turned off
an IP belonging to one of the entities that you mentioned in your

In general, I would approach the bizdev folks at Google about how to
send automated queries to Google with permission. Failing that, be
aware that if a tool sends too many queries to Google, we do reserve
the right to disable the IP address(es) of that tool. One thing I
would *not* recommend is that if a tool is blocked for bad behavior,
trying to make the tool more "sneaky" (e.g. trying to make the tool
look closer to a web browser). Attempts to fake out Google and pretend
to be more like a web browser (after you've been blocked once already)
is an example of the sort of thing that is really bad in our opinion. “

JohnMu further stated:

“Hi Scott and a belated welcome to the groups!

This is an interesting topic. I'm not quite sure how all those links
apply to our terms of service, which do not allow these kinds of
automated tools. I wrote a bit about tools like that a short while

A tool accessing other websites should try to obey the rules set forth
by that website. In general, these rules are described in several

  1. The robots.txt covers which URLs may be accessed and which ones are
    disallowed. You'll notice that in our robots.txt we explicitly
    disallow "/search", which is what most of the ranking tools generally
    try to access.
  2. The server result codes give more information when a URL is
    accessed. When our network recognizes automated queries, it may return
    a result code of 500 or similar.
  3. The HTTP headers returned by the server can provide information
    through the "x-robots-tag".
  4. A HTML page may provide information through a "robots" (or in our
    case, "googlebot") meta tag.

As far as I am aware, there are no "SEO-tools" that have permission to
access our web-search results in an automated way. I am also not aware
of any plans to change that in the near future.”

Navneet Kaushal

Navneet Kaushal

Navneet Kaushal is the founder and CEO of PageTraffic, an SEO Agency in India with offices in Chicago, Mumbai and London. A leading search strategist, Navneet helps clients maintain an edge in search engines and the online media. Navneet's expertise has established PageTraffic as one of the most awarded and successful search marketing agencies.
Navneet Kaushal
Navneet Kaushal
Most popular Posts
Upcoming Events
Events are coming soon, stay tuned!More