In a recent thread on SEW it has some juicy discussion related to a post on High Rankings. In this particular post Google Engineer, Dan Crow, Director of Crawl Systems at Google came up with some valuable tips and suggestions to get your site indexed by the most popular search engine. The tips are new and one can't find them at Webmaster Guidelines.
It was really interesting to find Google Engineers sharing secret suggestions on improving the ranking position of a site on Google. In fact Dan Crow even recommended some methods to get the site indexed. Dan Crow touched many topics in the discussion including:
- What Is Indexing?
- Robots.txt Explored
- Google Still Loves Its PageRank
- Did You Know About These Tags?
- Breaking News!
- Webmaster Central Tools
- Sitemaps Explored
- Flash and AJAX
- The Q&A
- The Dreaded Supplemental Results
Dan Crow started the discussion from a very basic level by making people know what Ă˘â‚¬Ĺ“indexingĂ˘â‚¬Âť means and how Google goes about it. Google crawler first looks at the robots.txt file in order to know where it should look for. He then proceeded to explain how to use the robots.txt file in the best possibe manner for excluding pages and directories from your site that you might not want indexed, like the cgi-bin folder. He told that there are over 200 factors, with Ă˘â‚¬Ĺ“relevanceĂ˘â‚¬Âť that have major roles to play. Dan Crow also discussed the importance of PageRank and that building a site with unique content for users is one of the best approaches to take.
The most important topic was the braking news "unavailable_after" tag that provides webmasters the ability to tell Google when to stop indexing a page at a certain time. This is very useful for people with e-commerce sites and are left with lots of expired coupons, older news items, and just about anything that is temporary and not permanent in nature on a site. Moreover, Nosnippet & Noarchive tags were also discussed by him. These tags are often not recommended by Google and the reason behind this according to him is "snippets are extremely helpful to visitors, as is showing the cache". As these tags eliminate some problems related to Google caching and improperly displays the snippets below the titles in the search results. But if one wants to use these tags, Google has no problem with this.