The recent Google webmaster video by Matt Cutts is a bit technical but quite insightful when we talk about the importance of HTTP Vary: User-Agent header.
The question asked was:
“What is Google's position about continuing to recommend the HTTP Vary: User-Agent header for specific mobile websites after big players like Akamai said they don't cache the URLs that include it? Would you still recommend using it?”
Cutts explained briefly what a cache is and how it is helpful in terms of content delivery. He said, if you ask for a webpage and then you ask for the exact webpage 3 seconds later, why do all the work to recompute what the webpage might look like if you are doing a dynamic sort of url, when you save that content and return the same content to the web users. You might have to recompute the data once in an hour or a day. But, once you have done the work on how a page should look like, of the times you can cache it. So, cache is helpful as it helps reduce the load on webservers.
But the problem arrives when the user is doing the query from the desktop and sometimes the user is doing the mobile site. You need to cache the desktop site and when the user asks for the same from a smartphone, for them to get the desktop page when he should get the smartphone page back. You can handle cache by handling the HTTP Vary User-Agent Header. It says that if you are a cache, whether you are on the server site or you are ISP or CDM, there are ways you can specify what to save to cache. And if there's an http header that says that things can vary by user agent like chrome versus mobile smartphone user agent then cache in theory can determine up to how many copies are to be saved as there will be a number of different user agents.
Taking the example of Akamai, Matt said, if there are 600 different user agents, there are chances that Akamai don’t want to save 600 different copies of the page. Therefore, Akamai has said that it would not cache the page that carried HTTP Vary header, if it is only by user agent. The reason being, there can be so many different versions of page and Akamai might be worried of either using the wrong version or using a lot of storage space.
But, Google suggests people to use HTTP Vary headers because of the lots of opportunities where server side caches, ISP caches, and other caches can use the information and do something really smart to avoid extra work or competition. Matt further said that individual content delivery network might decide upon how they would treat that specific header, but it is always a good idea to have a website that sends information along. This would lead to various caches between you and the user do something smart. If we take that out completely, there would be no information available and servers would not be able to handle that.
Matt also said that in terms of Google indexing, it does not appear to Google whether a URL can return different content on the basis of user agent. Therefore, Google looks at the HTTP Vary Header because it helps in assessing whether the page might be targeted both to; the desktop and mobile. So, that’s another reason to use it. Google recommends people to use it because it will help Google figure out if the particular URL can return different content to desktop and mobile.
At Last:
Although, the use of HTTP Vary: User-Agent header is complicated issue but Google recommends that you should continue using it if you are serving content to different desktop versions, smartphones, or something like that.
Have a look at the video:
More videos: http://www.youtube.com/GoogleWebmasterHelp
Webmaster Central Blog: http://googlewebmastercentral.blogspot.com/
Webmaster Central: http://www.google.com/webmasters/
