Apr 16, 2008 118 reads by Navneet Kaushal

Members at Webmaster World discusses about the problem of server header status codes. It is one of those topics that aren't touched often!

According to the Webmasters World thread, almost every server can return an incorrect HTTP status in the server header. But, Microsoft IIS (Internet Information Service) has a particularly long history of such errors been shown. For example:

Tried to set up a "custom 404" page, so your users get a friendly human like message.

When a bad URL is requested, the server responds with a 302 redirect to the custom error page (often called 404.html for example). Then the custom message is returned with a 200 OK http status.

The user agent is never informed by the server that the status of the requested URL should be 404. As a result a '302 temporary redirect' is issued. Unless the server informs the crawler about the correct status of requested URL, it won't matter what error code is written in the web page. Crawlers (including Googlebot) communicate with servers and not individual web pages.

Some of the interesting threads at Webmasters World:

"Any bad URL can be indexed according to the standard handling of an internal 302 redirect:

a) The content of the redirect's target URL is indexed.

b) With the original URL as the location.

So the bad URLs can start piling up as duplicate URLs for the same exact content."

Microsoft's .NET platform and IIS are particularly susceptible to this kind of errors. Inspite of the fact that the default error handling returns a correct 404 status code, still it is the way by which custom error messages are set up that pose the problem.

"For the IIS user, there is one other caution I should mention about 404 handling. If you are using .NET, then there are two levels of error handling: at the IIS level and at the .NET level. It is also common to find that only one of these two levels is set up correctly. So when you're stress testing your URLs, try a bad URL with an .asp (.aspx) extension, and also try a bad url with a .htm extension.

I felt it was important to focus on this issue in a dedicated thread – it's coming up much too often, both in threads here and in the sites of new clients that I evaluate.

The problem is by no means confined to IIS, either. For example, I see it on Apache servers running Tomcat for .jsp pages."

Navneet Kaushal

Navneet Kaushal

Navneet Kaushal is the founder and CEO of PageTraffic, an SEO Agency in India with offices in Chicago, Mumbai and London. A leading search strategist, Navneet helps clients maintain an edge in search engines and the online media. Navneet's expertise has established PageTraffic as one of the most awarded and successful search marketing agencies.
Navneet Kaushal
Navneet Kaushal
Most popular Posts
Upcoming Events
Events are coming soon, stay tuned!More