Google, Microsoft & Yahoo! enable Cross-submissions via robots.txt on Sitemaps.org

Feb 28, 2008 | 3,834 views | by Navneet Kaushal
VN:F [1.9.20_1166]
Rating: 0.0/5 (0 votes cast)

In 2006, Google, Microsoft Live Search and Yahoo! came together and extented their joint support for the Sitemaps protocol. So now furthering this team play, Google, Microsoft and Yahoo! have announced the Cross-host submission for Sitemaps.

logos Google, Microsoft & Yahoo! enable Cross submissions via robots.txt on Sitemaps.org

The Cross-host submission will make it easier for webmasters to manage their Sitemap submissions to the major search engines. With this announcement, webmasters can now submit Sitemaps that correspond to several differently-hosted websites using a single mechanism. For background, a Sitemap file contains the URLs for the pages on a site along with meta-data, such as priority, last crawled date and changed frequency for the content. To ensure validity of this metadata, Sitemaps have previously been required to be on the same host and path as the URLs they contain. This requirement forced the Sitemaps files to be hosted on the same servers as the actual site content.

With today's announcement, a Sitemap can now be hosted on a different host and path than the URLs it contains. For example, say you have a Sitemap (sitemap-www.xml) for the URLs on http://www.example.com but you want to put that Sitemap on http://sitemaps.example.com. That is now possible. To make the Sitemap valid and preserve data security you need to refer to it from the robots.txt file on the site where the URLs it contains are located. For example, add the following link to http://www.example.com/robots.txt:

Sitemap: http://sitemaps.example.com/sitemap-www.xml

The catch is that all the URLs in the sitemap file need to be within the same domain as the robots.txt file (i.e. moneycentral.msn.com/* in this example). This applies equally for sitemap index files and for compressed files. The protocol is offered under an Attribution-ShareAlike Creative Commons License, so it can be used by any search engine, derivative variations using the same license can be created and it can be used for commercial purposes.

With this team play, we believe that their next move together would be to develop meaningful standard support for robots.txt files. But that only time will tell, however for now it is the moment to celebrate, as at least these competitors together are agreeing on something. What do you say?

4.thumbnail Google, Microsoft & Yahoo! enable Cross submissions via robots.txt on Sitemaps.org

Navneet Kaushal

Navneet Kaushal is the founder and CEO of PageTraffic, an SEO Agency in India with offices in Chicago, Mumbai and London. A leading search strategist, Navneet helps clients maintain an edge in search engines and the online media. Navneet's expertise has established PageTraffic as one of the most awarded and successful search marketing agencies.
4.thumbnail Google, Microsoft & Yahoo! enable Cross submissions via robots.txt on Sitemaps.org
4.thumbnail Google, Microsoft & Yahoo! enable Cross submissions via robots.txt on Sitemaps.org