Apr 15, 2008 113 reads by Navneet Kaushal

Yahoo! announced the latest update to its Search Crawler called Yahoo Slurp 3.0. Although the Slurp 3.0 will recognize the same parameters and all robots.txt directives for Yahoo! Slurp, still it will be registered as Slurp 3.0 in the user web logs.

As this new software is being slowly introduced, users would have an introduction to the following changes:

The crawlers will originate from the crawl.yahoo.net.domain as usual, but they would start crawling from a different and much smaller set of IP addresses. As before, reverse DNS checks performed to identify Yahoo! crawlers will still be working. According to Yahoo!,“Please note that if you're using IP-based recognition of our crawlers, you might see a drop in crawl/coverage from Yahoo! We strongly recommend that you move to 'reverse DNS based identification of Yahoo! Slurp if you're using any other method to avoid this problem. The current set of IPs will disappear from your web logs in the next several weeks.”

As mentioned above, the new user-agent Yahoo! Slurp 3.0 will follow existing robots.txt directives for 'Slurp' or 'Yahoo! Slurp. However, directives specific to Yahoo! Slurp 2.0 will not be recognized or followed by the new crawler. Therefore, it is recommended by Yahoo! that the user specifies the shorter version of User-agent: Slurp.

These new changes will be only applicable to the main Yahoo! Web Search Crawlers and not would not affect specific search crawlers. For example, Yahoo! Slurp China and others.

Navneet Kaushal

Navneet Kaushal

Navneet Kaushal is the founder and CEO of PageTraffic, an SEO Agency in India with offices in Chicago, Mumbai and London. A leading search strategist, Navneet helps clients maintain an edge in search engines and the online media. Navneet's expertise has established PageTraffic as one of the most awarded and successful search marketing agencies.
Navneet Kaushal
Navneet Kaushal
Most popular Posts
Upcoming Events
Events are coming soon, stay tuned!More