Based on a study of 7,500 Web sites conducted by Penn State University, Google emerged as Webmasters favorite. C. Lee Giles, the David Reese Professor of Information Sciences and Technology who led the research team which developed a new search engine—BotSeer—for the study, found the result "surprising".
Giles said, "We expected that robots.txt files would treat all search engines equally or maybe disfavor certain obnoxious bots, so we were surprised to discover a strong correlation between the robots favored and the search engines’ market share".
Some of the results are:
More than any other search engine, Google's is most favored and called most often.
Take a look at the chart below and guess who the prodigal one is.
Apparently, the study is titles, "Determining Bias to Search Engines from Robots.txt (PDF)," Barry Schwartz has a local copy for easy download.
Recommend this story
Navneet Kaushal, CEO PageTraffic is a trusted authority in the search engine marketing industry. He is a featured author at Web Pro News, Search Newz, Promotionworld, Website Notes, DevWebPro, SEO Article and Web Help Now among many others.
Follow Navneet Kaushal on Google +.
@pagetraffic
