During the weekend, Yahoo!'s social bookmarking property del.icio.us seems to have prevented robots of other search engines including Google from spidering the website or fishing out fresh web pages, bookmarks and websites, observes Collin.
It was clear that it was not a simple robots.txt exclusion, rather it was a 404 response that was being shown based on who the requesting User-Agent was.
An observation was made that when the User-Agent was set to Googlebot the 404 error showed up each time, and the same response was received when any page other than the homepage was attempted to be navigated.
"I took a look at del.icio.us' robots.txt," says Collin "and found that it was disallowing Googlebot, Slurp, Teoma, and msnbot for the following:
- Disallow: /inbox
- Disallow: /subscriptions
- Disallow: /network
- Disallow: /search
- Disallow: /post
- Disallow: /login
- Disallow: /rss"
Why on earth Yahoo! may have done this, competition or what. As of now Yahoo has added delicious in its search pages, and that delicious is an important property for Yahoo to bank upon. Apparently, it has begun to make use of its powers from blocking other search engines from taking advantage from spidering the information contained therein.
Moreover, Yahoo! also has an edge over Ask, Google and MSN and that is, it's able to notice the web pages, etc. through human bookmarking, tagging etc. on Delicious, even before other search engine can. Consequently, it leads to better search results and rankings.
In this case Yahoo seems to have made use of its discretionary powers to combat competition. While preventing others from examining your business secrets is a common strategy used by business, it is quite a bold move from the company. What are your thought about it?
There's a discussion about it at Spinn
Recommend this story
Navneet Kaushal, CEO PageTraffic is a trusted authority in the search engine marketing industry. He is a featured author at Web Pro News, Search Newz, Website Notes, DevWebPro, SEO Article and Web Help Now among many others.
@pagetraffic
{ 5 comments… read them below or add one }
Hi There Navneet,
I just wanted to add two cents of something i noticed over the weekend as well. I did some searching on Yahoo on Sunday (over the same time period) and noticed that Yahoo was inserting the number of delicious “bookmarks” into their search results. For example, when i did a search for “business blogs”…i noticed that some listings actually had the number of bookmarks in delicious by their listing. So o think its fair to say that they have been experimenting in feeding delicious bookmarks into their search results to offer more comprehensive search results….much like Google’s universal search. However i noted that our site “iBlogbusiness.com” (which has several delicious bookmarks) did not have the little icon by the search results. So im not sure how they are determining who they show the bookmark icon to at the moment, but it will be interesting to see what yahoo decides to roll out:)
Best Regards,
Christian
Who cares if delicious will block google or not?
In case you never noticed, all Bookmarks added to delicious have rel=”nofollow” set in them,
meaning Google wont/cant/shouldn’t use delicous bookmarks to “discover” new sites to index.
The directories being blocked are not important for search and should not get spidered by Google.
E.g. Google does not need to spider the “post a new url” page.
The actual link pages for all users are still spiderable. Of course, as Steve notes, they include rel=”nofollow”, and so are of little use to Google.
As a seocontest2008 participant I am very interested about the social bookmarking issues.
Your news about Yahoo’s new move about del.icio.us are really intriguing. Thank you very much for such valuable information. I can’t do much about Yahoo’s politics, I only have the power of choice to use or not to use del.icio.us services. For the present seocontest2008 purpose I am interested to gain good position on Google, and not on other search engines. If Yahoo has begun to make use of its powers from blocking other search engines from taking advantage from spidering the information contained therein, then I probably will not be interested to use Yahoo’s social bookmarking services, hence there are so many other options available.
We’ve put up a post on our site just to explain why this is actually wrong:
http://www.accuracast.com/search-daily-news/seo-7471/false-reports-about-yahoo-blocking-googlebot-on-delicious/