Nov 29, 2006 672 reads by Navneet Kaushal

Remember, a couple of months ago Google told us how to verify Googlebot. It explained verification requires a reverse DNS look up which is followed by a forward DNS lookup.

Microsoft has introduced similar verification method for MSNBot. Live Search's Weblog explains: "anyone could call themselves ‘MSNBot’, and proceed to be as rude and aggressive as they like. Fortunately, there is a way you can catch these impersonators. Here is how it works:

1. When you get a page view request, it specifies a user-agent and an IP address. As I described above, all requests from Live Search use a user agent starting with the word ‘MSNBot’.

2. If you see the MSNBot user-agent, it’s time to check the identity of the bot. Starting with the IP address (i.e., you can use reverse DNS lookup to find out the registered name of the machine.

3. Once you have the host name (in this case,, you can check that it really is coming from Live Search. The name of all live search crawlers will end with ‘’. If the name doesn’t end with ‘’, you know it’s not really our crawler.

4. Finally, you need to verify that the name is accurate. In order to do this, you can use Forward DNS to see the IP address associated with the host name. This should match the IP address you used in Step 2 – if it doesn’t, it means the name was fake.

By verifying the crawler’s identity, you can catch masquerading crawlers. When you do catch one, you can simply return an HTTP Error, thus blocking them from seeing your content."

Navneet Kaushal

Navneet Kaushal

Navneet Kaushal is the founder and CEO of PageTraffic, an SEO Agency in India with offices in Chicago, Mumbai and London. A leading search strategist, Navneet helps clients maintain an edge in search engines and the online media. Navneet's expertise has established PageTraffic as one of the most awarded and successful search marketing agencies.
Navneet Kaushal
Navneet Kaushal
Most popular Posts
Upcoming Events
Events are coming soon, stay tuned!More