Matt Cutts, Google Head of Search Spam Matt Cutts, has explained how Google evaluates which new search algorithms to use and which to refine in his latest Webmaster Help video.
He was replying to a question posed by James Foster of Sydney, Australia –
“What are some of the metrics that Google uses to evaluate whether one iteration of the ranking algorithm is delivering better quality results to users than another?”
Matt stated that whenever an engineer begins evaluating a new search quality change, and they need to find whether it is an improvement, what helps is the presence of hundreds of quality raters, with experience in rating URLs as good or bad, spam and the sorts of similar things.
When there is a change, the flux is visible. One can find what has moved up and what down. The team can look at example searches where the results changed a lot. They can find if in the changed search results, the URLs that moved up were those typically higher rated by the search quality raters compared to the URLs that moved down.
Google has already got a stored data bank of the ratings. However, sometimes, there are things that haven’t been rated. These URL’s are sent out to the raters to get the results of either a side-by-side. This helps in deciding which of the search results is better. Engineers can find which URL is good and which is spam, enabling them to assess if they are really progressing.
If Google is getting close to launch a new algorithm, they may launch what they term a ‘live experiment’. The two different algorithms are applied simultaneously to generate search results, which may be interweaved as well. If the newer set of search results gets more clicks, you may conclude that the results produced by this algorithm may be better than the other.
However, Google accepts that the process is not foolproof. People may click more on spam, thus skewing the metrics and creating the false impression that the new algorithm is not all that successful. The ratings can’t be accepted blindly and nothing replaces the judgment of the quality launch committee.
Google does have a lot of metrics to test the search results. However, at the end, nothing replaces the judgment of a human team.
Matt Cutts Explains How Google Evaluates New Search Algorithms!,