There is a lot that goes on in the Google office, before the search engine giant gives the thumbs up to a particular algorithmic change. In the recent Webmaster Help video, Google's head of Search Spam, Matt Cutts talks about how Google analyzes the quality of an algorithm change before deciding whether or not to launch it.
The question posed was:
“What are some of the metrics that Google uses to evaluate whether one iteration of the ranking algorithm is delivering better quality results to users than another?”
In the relatively lengthy video, Matt reveals that Google uses numerous quality raters to evaluate whether or not the new version of an algorithm is an improvement over the current version. These raters have been used so far to rate URLs as good, bad or spam. Google analyzes this database of URL ratings to evaluate if an algorithm tweak has created positive results. Then depending on these ratings, Google can calculate if more ‘good’ URLs are displayed in the search results than the ‘bad’ or spammy ones.
Yet another thing which Google does, explained Matt, is compare the search results, simultaneously to evaluate which algorithm reverts back the best quality. Once Google is done with a few trial rounds, it launches a live experiment before releasing the new algorithmic change to all the users. In the live experiment, Google serves both the current and the new version to different sets of users. Based on the number of clicks to sites within each, Google determines which version is preferred by the users.
However, these metrics are not the ultimate factors deciding the launch of any particular algorithm, as in Matt's own words, “Nothing replaces your judgment and the judgment of the quality launch committee.”
Matt says that when the quality raters are rating URLs they look for things which have been rated as “relevant” or “spam” by users, apart from the overall quality of the page. Google's metrics then blends all these things together to determine if the quality of search results improved or not. According to Matt, Google also takes into account different countries and different languages and pays special attention to countries from where more spam comes. Even if users keep on clicking on spam results unknowingly, the web spam team can flag these spam results using the quality raters during the live experiments. Google keeps on updating and revising the quality rating guidelines frequently to make sure that the algorithmic evaluation goes on well.How Google Tests its New Algorithm Before rolling it out to the Audiences, Explains Matt Cutts!,