Control The Indexing And Accessing Of Your Sites By Search Engines

Jan 27, 2007 | 2,818 views | by Navneet Kaushal
VN:F [1.9.20_1166]
Rating: 0.0/5 (0 votes cast)

A post on Google blog gives important details to the web publishers about how they can control indexing and accessing of sites by search engines and Google itself. The most important tool in this regard is the robots.txt file. Robots.txt file gives powerful control to site owners on how the site is searched. The post reads “you may have a few pages on your site you don't want in Google's index. For example, you might have a directory that contains internal logs, or you may have news articles that require payment to access. You can exclude pages from Google's crawler by creating a text file called robots.txt and placing it in the root directory. The robots.txt file contains a list of the pages that search engines shouldn't access. Creating a robots.txt is straightforward and it allows you a sophisticated level of control over how search engines can access your web site.”

Besides the robots.txt file there is robots META tag by which you can gain more fine control over the individual pages. This requires specific META tags to HTML pages giving you the control over the way individual page is indexed.

4.thumbnail Control The Indexing And Accessing Of Your Sites By Search Engines

Navneet Kaushal

Navneet Kaushal is the founder and CEO of PageTraffic, an SEO Agency in India with offices in Chicago, Mumbai and London. A leading search strategist, Navneet helps clients maintain an edge in search engines and the online media. Navneet's expertise has established PageTraffic as one of the most awarded and successful search marketing agencies.
4.thumbnail Control The Indexing And Accessing Of Your Sites By Search Engines
4.thumbnail Control The Indexing And Accessing Of Your Sites By Search Engines