Google Announces New Googlebot user-agent for Crawling & Indexing Smartphone Content!

Jan 24, 2014 | 1,211 views | by Navneet Kaushal
VN:F [1.9.20_1166]
Rating: 5.0/5 (1 vote cast)

Google has always used different feature & smartphone specific crawlers referred to as Googlebot-Mobile to index & crawl content. Since, smartphones and feature phones have different device capabilities, webmasters unknowingly block the smartphone indexing or crawling, when they just wanted to blog a feature phone indexing or crawling. As a result, Google started finding it difficult to index smartphone content of some sites or to even recognize that these websites were smartphone optimized.

Google has now decided to say goodbye to "Googlebot-Mobile" for smartphones in 3-4 weeks' time. After that, the user-agent for smartphones will be simply identified as "Googlebot". It will still list "mobile" elsewhere in user-agent string.

The Google Webmaster Central blog also shared the new and old user-agents for smartphones.

The new Googlebot:

Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Googlebot-Mobile that will be retiring soon:

Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

The change will only affect Google-bot Mobile for smartphones. The remaining two Google-bot Mobile crawlers will still refer to feature phones in their user-agent strings. These are:

  • SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
  • DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

On testing the site using Fetch as Google feature in Webmaster Tools, you will see a full list of existing crawlers of Google in Help Center.

Crawling and Indexing

The new Googlebot for smartphones crawler will follow:

  • robots.txt
  • robots meta tag
  • HTTP header directives

When the new crawler is deployed, robots.txt directive will block crawling by Google's feature phone crawlers:

User-agent: Googlebot-Mobile
Disallow: /

Google has analyzed that the update will affect less than 0.01% of URLS. It will also give webmasters more control over indexing & crawling of their content.

Google Announces New Googlebot user-agent for Crawling & Indexing Smartphone Content!, 5.0 out of 5 based on 1 rating
1.thumbnail Google Announces New Googlebot user agent for Crawling & Indexing Smartphone Content!
Navneet Kaushal is the founder and CEO of PageTraffic, an SEO Agency in India with offices in Chicago, Mumbai and London. A leading search strategist, Navneet helps clients maintain an edge in search engines and the online media. Navneet's expertise has established PageTraffic as one of the most awarded and successful search marketing agencies.
1.thumbnail Google Announces New Googlebot user agent for Crawling & Indexing Smartphone Content!
1.thumbnail Google Announces New Googlebot user agent for Crawling & Indexing Smartphone Content!