Nov 16, 2007 114 reads by Navneet Kaushal

Based on a study of 7,500 Web sites conducted by Penn State University, Google emerged as Webmasters favorite. C. Lee Giles, the David Reese Professor of Information Sciences and Technology who led the research team which developed a new search engine—BotSeer—for the study, found the result "surprising".

Giles said, "We expected that robots.txt files would treat all search engines equally or maybe disfavor certain obnoxious bots, so we were surprised to discover a strong correlation between the robots favored and the search engines’ market share".

Some of the results are:

More than any other search engine, Google's is most favored and called most often.

robots-txt-1.jpg

Take a look at the chart below and guess who the prodigal one is.

robots-text2.jpg

Apparently, the study is titles, "Determining Bias to Search Engines from Robots.txt (PDF)," Barry Schwartz has a local copy for easy download.

Navneet Kaushal

Navneet Kaushal

Navneet Kaushal is the founder and CEO of PageTraffic, an SEO Agency in India with offices in Chicago, Mumbai and London. A leading search strategist, Navneet helps clients maintain an edge in search engines and the online media. Navneet's expertise has established PageTraffic as one of the most awarded and successful search marketing agencies.
Navneet Kaushal
Navneet Kaushal
FOLLOW US AND GET LATEST NEWS
125,891
Most popular Posts
Tweets
Upcoming Events
Events are coming soon, stay tuned!More