Google Accepts The "noindex" Directive In The robots.txt, Temporary or Permanent?

Nov 22, 2007 | 13,588 views | by Navneet Kaushal
VN:F [1.9.20_1166]
Rating: 0.0/5 (0 votes cast)

In a post called 'Validate your robots.txt – Googlebot becomes smarter' Sebastian writes that "forgotten –and, until recently, ignored– statements in your robots.txt might change the crawler’s behavior all of a sudden, without notice".

In this regard, John Müller from Google Zürich answers:

"At the moment we will usually accept the "noindex" directive in the robots.txt, but we are not yet at a point where we are willing to set it into stone and announce full support. … I just want to remind everyone again that this is something that may still change over time. Be careful when playing with things like this."

Like all things important, it's not of it's happening, it's when it'll happen that matters.

4.thumbnail Google Accepts The noindex Directive In The robots.txt, Temporary or Permanent?

Navneet Kaushal

Navneet Kaushal is the founder and CEO of PageTraffic, an SEO Agency in India with offices in Chicago, Mumbai and London. A leading search strategist, Navneet helps clients maintain an edge in search engines and the online media. Navneet's expertise has established PageTraffic as one of the most awarded and successful search marketing agencies.
4.thumbnail Google Accepts The noindex Directive In The robots.txt, Temporary or Permanent?
4.thumbnail Google Accepts The noindex Directive In The robots.txt, Temporary or Permanent?