Ad Testing- Research & Findings: SES, San Jose 2007

Aug 22, 2007 | 2,183 views | by Navneet Kaushal
VN:F [1.9.20_1166]
Rating: 0.0/5 (0 votes cast)

Have you been doing deep testing of your ad campaigns? Fear not — the panel in this session has and will be sharing tips you might want to try yourself. How important are headlines, descriptions and other elements in your control? How do consumers interact with the ads?


  • Gordon Hotchkiss, President & CEO, Enquiro Search Solutions Inc.


  • Gordon Hotchkiss, President & CEO, Enquiro Search Solutions Inc.
  • Anton E. Konikoff, Founder and CEO, Acronym Media
  • Jonathan Mendez, Founder & Chief Strategy Officer, OTTO Digital
  • Nico Brooks, Director of Search Technology, Atlas
  • Clay Bavor, Product Manager, Ads Quality, Google

Beginning the discussion today is Anton E. Konikoff the founder and CEO of Acronym Media.

Anton stresses on "keyword driven marketing".

Significance of "keyword driven marketing".

  • Customers are equated with keywords.
  • Pay attention to the language the customer uses to understand the meaning and write messages based on that.
  • While testing, remember the customer's voice and intent.
  • Pay equal weight to B-B and B-C.

Why perform Ad Testing?

  • You will receive quantifiable results, i.e.gain awareness within a controlled environment.
  • Classify the exact products and offerings.
  • Recognize the response of the message.

Key Metrics

  • Click To Rate – To be used as both a process and a final metric.
  • Conversion Rate by Ad – This is the final yardstick of success.

How to set up an ad test?

  • The numbers of ads one should test
  • Test as many as you are comfortable with.
  • Sample size
  • Amount of data
  • 100 clicks will do but 1000 is definitely better.

Ad Copy Testing

  • Run a test of the headings and URLs on display.
  • Ensure your headings remain consistent and also test the content description. After which, test the headline again.
  • Test the ad for call to action on whether it is on price point?
  • The one that performs best should be tested once again for description test and then test headline.

Run more granular tests:

  • Punctuation
  • Capitalization
  • Proper Case vs. Sentence Case
  • Accent Usage
  • Keyword Insertion

Anton showed a case study of a company called Sirius Radio. In an ad test they tested $30 rebate vs. One Month Free for one month. Finally, the figures were $30 rebate conversion and 3.5%, free month 1.5%.

Common Pitfalls of ad testing are:

  • High CTR may be detrimental.
  • Is the traffic qualified
  • Are too many variables used?
  • Stay aware of the changes that caused the impact.

Important tips and suggestions:

  • In the beginning use the simplest test.
  • With time, add more variables.
  • Anton suggests to be brave and gutsy in trying out new hypothesis. Challenge the status quo.
  • Set aside a budget specifically for testing.
  • Use a testing procedure that is ongoing and consistent. Find the right mix of elements and continue to push testing.
  • Continually test something or the other.
  • Do not settle for current results. Improvement is always possible through structured testing.

Next up is Nico Brooks from Atlas who gives suggestions and shares what she has learnt on ad testing.


  • Differentiate between branded and non-branded keywords.
  • Know the history of the ad clicks/interaction that lead to a conversion.
  • Pay more attention than that of a last click.
  • Defining route knowledge – it's how we think about where things are. Know where something is (like a business) without really knowing the exact physical location.

Big Idea

  • A lot of search activity is motivated by route know how.
  • Analogy – Ants and Compost Bucket
  • Ant problem at commune. So long as the bucket was moved, the ants would not get to it.
  • The ants used route knowledge to find the bucket and when it moved, the ants couldn't find it.
  • How we think about where something is has to do with how we get there.

Nico suggests that in terms of applying ad testing to search data remember that users know how they got where they got. Although they may not remember the name of the website they know the process and will try the same procedure.

A search study was conducted on a segmented 275k clicks starting from the first visit and reports as well as branded vs. non-branded.


  • Repeat visits were nearly % of total clicks.
  • Brand data was more than 50% of clicks
  • Repeat and Branded was nearly 1/3 of all clicks.
  • Repeat plus branded accounted for nearly 70% of clicks

Next up was Johnathan Mendes of OTTO Digital who is making his second appearance since yesterday's Ads In A Quality Score World Session.

Techniques to successful ad testing:-

  • Always have a single question to answer.
  • The best test will answer that question and ask many more.
  • Maintain a big difference between elements that you are testing.
  • Draw out the differences
  • Temporal Behavior
  • Results during the week may be different than those over the weekend
  • Amass enough data to reduce margin of error and increase confidence.
  • Too little data may induce false positive.
  • Segment results
  • Act! Take action on results and keep on testing.

Reasons for continuous tests

  • Testing is more reliable and far better than relying on guesstimates.
  • Testing is done because no matter how much we know there is always room to learn more.
  • Do not make conjectures.

Ad copy testing itself can yield a significant change in conversion.

  • Why do ads work?
  • Leaves an Impression on the user
  • Good advertisements create interest in the user about the product/service on offer
  • With expectation one finds what one wants
  • Persuasion The ulterior motive is to sell the product/service.

Test Ads by Match Type

This created unworthy ad groups and tested against each other based on their match type.

One should understand, test and measure the impact on your campaigns.

Brand Keywords – waste of money?

Tested 1 week on 1 week off

No seasonality.

Ad went to homepage

Single version of the ad

30 branded keywords

Did they drive traffic? NO.

Did having the ads increase conversion? YES – up to 32%

Having the ads was a huge win.

Next in line to carry forth the discussion was Gordon Hotchkiss the President & CEO of Enquiro Search Solutions Inc.

What's Next?

  • Search results page has been pretty static the last several years. We are introducing many new variables to testing so a heads up is probably better.
  • Pace of change on the SERP is going to increase.
  • On the cusp of search experience changes

Gord says that we should try to create equilibrium between advertisements and organic result relevance. Also, we should improve advertisements in oreder to create a richer experience.

Quality of videos, images, video in addition to text will help us in target keywords through personalization better.

The next question that arises is whether users will stop clicking ads if natural results improve?

“When an image is introduced on a page, users in eye tracking study used the image to divide the page. Users looked at result immediately adjacent to the image. Scanning ran down from the image (in slot 3-4) which is different from what is seen in non-image based search results.” users are now navigating complicated webpages very quickly. This will effect advertising.

Last to address the session was Clay Bavor, Product Manager, Ads Quality, Google who was also making his second appearance since yesterday's Ads In A Quality Score World Session.

Clay stated that it was all about user experience and if the user was happy he would return.

And ad testing is crucial to ensuring good user experience. Some of Clay's pointers include:

  • “Step back and look at keywords. Are they conceptually grouped?
  • Don't drop the ball on the landing page. Take some time and make sure they correspond to the ad text and keywords that are driving traffic there.
  • Think of ad testing as putting yourself in the user's shoes to drive high quality traffic.
  • Check out "Search Query Report" in Adwords.
  • You can find a summary of keywords that are driving traffic which will help you improve the overall performance of campaigns and ad groups
  • Circle back to the user experience
  • Target effectively for the user experience your users want
  • Keep the users in mind
  • Test and re-test”

SOURCE: Search Engine Round Table.

4.thumbnail Ad Testing  Research & Findings: SES, San Jose 2007

Navneet Kaushal

Navneet Kaushal is the founder and CEO of PageTraffic, an SEO Agency in India with offices in Chicago, Mumbai and London. A leading search strategist, Navneet helps clients maintain an edge in search engines and the online media. Navneet's expertise has established PageTraffic as one of the most awarded and successful search marketing agencies.
4.thumbnail Ad Testing  Research & Findings: SES, San Jose 2007
4.thumbnail Ad Testing  Research & Findings: SES, San Jose 2007