Google has always used different feature & smartphone specific crawlers referred to as Googlebot-Mobile to index & crawl content. Since, smartphones and feature phones have different device capabilities, webmasters unknowingly block the smartphone indexing or crawling, when they just wanted to blog a feature phone indexing or crawling. As a result, Google started finding it difficult to index smartphone content of some sites or to even recognize that these websites were smartphone optimized.
Google has now decided to say goodbye to "Googlebot-Mobile" for smartphones in 3-4 weeks' time. After that, the user-agent for smartphones will be simply identified as "Googlebot". It will still list "mobile" elsewhere in user-agent string.
The Google Webmaster Central blog also shared the new and old user-agents for smartphones.
The new Googlebot:
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot-Mobile that will be retiring soon:
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
The change will only affect Google-bot Mobile for smartphones. The remaining two Google-bot Mobile crawlers will still refer to feature phones in their user-agent strings. These are:
- SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/18.104.22.168.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
- DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
Crawling and Indexing
The new Googlebot for smartphones crawler will follow:
- robots meta tag
- HTTP header directives
When the new crawler is deployed, robots.txt directive will block crawling by Google's feature phone crawlers:
Google has analyzed that the update will affect less than 0.01% of URLS. It will also give webmasters more control over indexing & crawling of their content.Google Announces New Googlebot user-agent for Crawling & Indexing Smartphone Content!,