Yahoo!'s New Robots.txt Tag To Block Unnecessary Content

May 4, 2007 | 1,537 views | by Navneet Kaushal
VN:F [1.9.20_1166]
Rating: 0.0/5 (0 votes cast)

Yahoo! has been making quite a few helpful improvements lately. The latest is robots-nocontent tag which will help webmasters differentiate main content and unrelated content for the Yahoo! crawler. Content in these restricted sections will not be used by Yahoo! for indexing the page or as excerpts in the SERPs.

Priyank Garg, project manager of Yahoo! Search informs, "This tag is really about our crawler focusing on the main content of your page and targeting the right pages on your site for specific search queries. Since a particular source is limited to the number of times it appears in the top ten, it's important that the proper matching and targeting occur in order to increase both the traffic as well as the conversion on your site. It also improves the abstracts for your pages in results by omitting unrelated text from search result summaries."

This is much like the Google’s no-follow tag and Yahoo! assures that it won’t be confused with cloaking. According to the Yahoo! Search Blog:

Note: Using a "nocontent" tag to mark explicit sections of content is not considered "cloaking" because all of the content on the page is available to protect the relevance of the results (unlike "cloaking" where we may be served content that is different from what visitors see).

The tag proves helpful to filter out navigation bars, advertising, boilerplate text and other content which is repeated across the site. The robots-nocontent does not affect the way links are treated.

4.thumbnail Yahoo!s New Robots.txt Tag To Block Unnecessary Content

Navneet Kaushal

Navneet Kaushal is the founder and CEO of PageTraffic, an SEO Agency in India with offices in Chicago, Mumbai and London. A leading search strategist, Navneet helps clients maintain an edge in search engines and the online media. Navneet's expertise has established PageTraffic as one of the most awarded and successful search marketing agencies.
4.thumbnail Yahoo!s New Robots.txt Tag To Block Unnecessary Content
4.thumbnail Yahoo!s New Robots.txt Tag To Block Unnecessary Content