Google Shows 'robots.txt file' in its SERPS! Can it Create Problems? Confusion Prevails!

May 16, 2008 | 4,721 views | by Navneet Kaushal
VN:F [1.9.20_1166]
Rating: 0.0/5 (0 votes cast)

There is a new concern at the Webmaster World with many Webmasters reporting that Google is showing their 'robots.txt file' in its SERPS (Search Engine Result Pages). Although, it shouldn't pose any threat, still there is considerable concern over this latest development. However, some other Webmasters are simply considering this display as an anomaly and believe that this won't be affecting websites and rankings in any ways.

Here are some interesting excerpts from the ongoing conversation at Webmasters World:

“Title says it all really. Any ideas why? It only seems to be affecting one of my sites. Can't see anything I might have done different with this one over the others.“

“From what I can see, Google shows 182,000 robot.txt files in their index – [inurl:robots.txt filetype:txt] is the query I was studying. By posting that specific query, I am making an exception to our usual policy of "no specific searches", but I feel it is generic enough – there are no keywords involved – to allow an exception in this case.

My guess is that somewhere, someone has linked to these robots.txt specific files – including yours, most likely. Certainly the top results for that query are robots.txt files that I know have been linked to – including ours here at WebmasterWorld. Google should simply drop all robots.txt files from their index, IMO – but we're the tail trying to wag the dog on this kind of thing. How is this causing you a problem, Mike? Is your robots.txt actually ranking on a keyword query?

At any rate, this does bring up the crazy question, how can you remove a robots.txt file from Google's index? If you use robots.txt to block it, that would mean that googlebot should not even request robots.txt – an insane loop. And of course, you don't use meta tags in a robots.txt file.

Unless this is some kind of a problem on keyword queries, I'd suggest you just chalk it up to strange and move on.”

“You never know if it is causing a problem with Google. Wondered if it was a side effect of something bad. We were the victim of a server hack (someone uploaded p*rn images into every page) two weeks ago, then last week someone linked to 1000+ pages that did not exist on the site (yes, Google indexed them too as duplicate content even though they had noindex/nofollow tags) so I wondered if this was the latest ranking attack by our competitors.“

“You can try adding this in your .htaccess file:

<filesmatch "robots\.txt">
Header set X-Robots-Tag "noindex, nofollow"

Then try again to remove with the Google Webmaster Tools "Remove URL".

4.thumbnail Google Shows robots.txt file in its SERPS! Can it Create Problems? Confusion Prevails!

Navneet Kaushal

Navneet Kaushal is the founder and CEO of PageTraffic, an SEO Agency in India with offices in Chicago, Mumbai and London. A leading search strategist, Navneet helps clients maintain an edge in search engines and the online media. Navneet's expertise has established PageTraffic as one of the most awarded and successful search marketing agencies.
4.thumbnail Google Shows robots.txt file in its SERPS! Can it Create Problems? Confusion Prevails!
4.thumbnail Google Shows robots.txt file in its SERPS! Can it Create Problems? Confusion Prevails!