Nov 12, 2007 114 reads by Navneet Kaushal

In a post called, "ASP.NET 2.0 Mozilla Browser Detection Hole," Brendan Kowitz writes that 'there is something drastically wrong with the way search engines have been indexing my ASP.NET 2.0 blog.'

Back in March 2006, Googlebot was altered and it sparked off a bug in ASP.NET webservers. With this, sites wouldn't be crawled properly. In its place, it would receive 500 Internal Server Errors, through which they considered certain webpages as unavailable, even if they are.

Brendan is not alone in this dilemma, many users is Webmaster World are also expressing over the bug, which they feel is Google's fault. Tedster offers a solution:

  • "I've been researching this for several clients on the .NET 2.0 platform, and those who use Helicon's ISAPI Rewrite for url rewriting do not seem to have a problem with googlebot's US string. I haven't zeroed in yet on which solutions do trigger the 500 error problem, or why. A number of .NET/IIS rewriting solutions are discussed on Scott Gu' blog (he's a developer for Microsoft) and only some of them have the problem.
  • I'd sugggest that anyone using .NET 2.0 install Firefox and the UserAgentSwitcher add-on so they can try a page from their site as Googlebot and discover if they get a 500 error. It's a quick health check-up that is well worth the little bit of time it takes. I also note that mikey158's problem "fixed itself" with no apparent changes on his server – so possibly the Google crawl team is taking some action here when they see too many 500 http errors. All I can imagine them doing, however, is re-crawling with a different user-agent string, one that does not include "Mozilla"."

Discussion continues at Webmaster World.

Navneet Kaushal

Navneet Kaushal

Navneet Kaushal is the founder and CEO of PageTraffic, an SEO Agency in India with offices in Chicago, Mumbai and London. A leading search strategist, Navneet helps clients maintain an edge in search engines and the online media. Navneet's expertise has established PageTraffic as one of the most awarded and successful search marketing agencies.
Navneet Kaushal
Navneet Kaushal
Most popular Posts
Upcoming Events
Events are coming soon, stay tuned!More