Jan 24, 2014 113 reads by Navneet Kaushal

Google has always used different feature & smartphone specific crawlers referred to as Googlebot-Mobile to index & crawl content. Since, smartphones and feature phones have different device capabilities, webmasters unknowingly block the smartphone indexing or crawling, when they just wanted to blog a feature phone indexing or crawling. As a result, Google started finding it difficult to index smartphone content of some sites or to even recognize that these websites were smartphone optimized.

Google has now decided to say goodbye to "Googlebot-Mobile" for smartphones in 3-4 weeks' time. After that, the user-agent for smartphones will be simply identified as "Googlebot". It will still list "mobile" elsewhere in user-agent string.

The Google Webmaster Central blog also shared the new and old user-agents for smartphones.

The new Googlebot:

Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Googlebot-Mobile that will be retiring soon:

Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

The change will only affect Google-bot Mobile for smartphones. The remaining two Google-bot Mobile crawlers will still refer to feature phones in their user-agent strings. These are:

  • SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/ (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
  • DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

On testing the site using Fetch as Google feature in Webmaster Tools, you will see a full list of existing crawlers of Google in Help Center.

Crawling and Indexing

The new Googlebot for smartphones crawler will follow:

  • robots.txt
  • robots meta tag
  • HTTP header directives

When the new crawler is deployed, robots.txt directive will block crawling by Google's feature phone crawlers:

User-agent: Googlebot-Mobile
Disallow: /

Google has analyzed that the update will affect less than 0.01% of URLS. It will also give webmasters more control over indexing & crawling of their content.

Google Announces New Googlebot user-agent for Crawling & Indexing Smartphone Content!, 5.0 out of 5 based on 1 rating
Navneet Kaushal
Navneet Kaushal is the Editor-in-Chief of PageTraffic Buzz. A leading search strategist, Navneet helps clients maintain an edge in search engines and the online media. Navneet's expertise has established PageTraffic Buzz as one of the well know digital marketing blog.
Navneet Kaushal
Navneet Kaushal
Most popular Posts
Upcoming Events
Events are coming soon, stay tuned!More