Search Engine Journal has an interesting report on DreamHost, a Web hosting & Server company who has asked its Client sites to block Googlebots. The news came known when a letter DreamHost sent one of its clients
Zoso emerged.
Stated in the letter from DreamHost:
“This email is to inform you that a few of your sites were getting hammered by Google bot. This was causing a heavy load on the webserver, and in turn affecting other customers on your shared server. In order to maintain stability on the webserver, I was forced to block Google bot via the .htaccess file.
[Limit GET HEAD POST
order allow,deny
deny from 66.249
allow from all]
You also want to consider making your files be unsearchable by robots and crawlers, as that usually contributes to high number of hits. If they hit a dynamic file, like php, it can cause high memory usage and consequently high load…â€
Readers commented on varying degrees. One said “This is a host everyone dreams of, a host that will ban search engine’s bots. Dreamhosts go back to stone age!â€. While a calmer one said, “There is too little information for anyone to do a proper analysis. With a dedicated server, I have occasionally had to tell Google and Yahoo! to stay out of my domains for as much as a couple of weeks. They can really hammer a server when they do a deep-crawl, which doesn’t happen often.A site that is getting deep-crawled may be too busy for shared hosting. The last shared hosting provider I had was kind enough to give me six weeks to find a dedicated plan once I’d outgrown their resources. I’ve known people whose hosting providers merely shut down their sites without so much as a heads up email saying, “You should get a new hosting provider immediatelyâ€. It sounds very much like growing pains for a successful site. Maybe it’s just time for them to take the next step and move to dedicated hosting.â€
This is not the first time DreamHost has manipulated (or tried to) client sites, as is evident in a forum at Digital Point.
Recommend this story
Navneet Kaushal, CEO PageTraffic is a trusted authority in the search engine marketing industry. He is a featured author at Web Pro News, Search Newz, Promotionworld, Website Notes, DevWebPro, SEO Article and Web Help Now among many others.
Follow Navneet Kaushal on Google +.
@pagetraffic
