In a post called 'Validate your robots.txt – Googlebot becomes smarter' Sebastian writes that "forgotten –and, until recently, ignored– statements in your robots.txt might change the crawler’s behavior all of a sudden, without notice".
In this regard, John Müller from Google Zürich answers:
"At the moment we will usually accept the "noindex" directive in the robots.txt, but we are not yet at a point where we are willing to set it into stone and announce full support. … I just want to remind everyone again that this is something that may still change over time. Be careful when playing with things like this."
Like all things important, it's not of it's happening, it's when it'll happen that matters.
Recommend this story
Navneet Kaushal, CEO PageTraffic is a trusted authority in the search engine marketing industry. He is a featured author at Web Pro News, Search Newz, Promotionworld, Website Notes, DevWebPro, SEO Article and Web Help Now among many others.
Follow Navneet Kaushal on Google +.
@pagetraffic
