Recently, Peter Norvig, the Director of Research at Google was asked. How does Google ensure that the quality of SERPs returned by it is good?
To which he replied "We test it in lots of ways." Firstly, "… we track what users are clicking on. If they click on the number-one result, and then they're done, that probably means they got what they wanted." But "If they're scrolling down, page after page, and reformulating the query, then we know the results aren't what they wanted."
A second way he mentions is to "… randomly select specific queries and hire people to say how good our results are. These are just contractors that we hire who give their judgment. We train them on how to identify spam and other bad sites, and then we record their judgments and track against that."
He realizes that while a "real opinion" is a "gold standard" but because its a costly affair it can be done within limits only. So in order to make up for the limits he says "We also invite people into the labs, or sometimes we go into homes and observe them as they do searches. It provides insight into what people are having difficulty with." You can read the full interview by Technology Review here.