Welcome to the Listening Post!

Building a Cost Effective, In-House Normative Concept Test Program

Tweet

 

Last week, I hosted a research webinar with one of our customers, Morgan Richards, senior manager of research with AutoTrader.com.  His topic was about bringing concept testing in-house.

AutoTrader.com provides classified ads for purchasing and selling automobiles within the United States. Traditionally, AutoTrader.com has relied heavily on usability testing, with UX (User Experience) analysts dedicated to live site testing and to determining what sort of impact site changes have on KPIs and downstream metrics. “That gives a great understanding of site behavior,” said Morgan, “but there was a gap in the testing process when it came to soliciting feedback early on in the process. We had no quantitative consumer feedback.”

A typical concept test follows these four steps:

1. Email users – Leveraging the database of prospects who have visited AutoTrader.com, the marketing department creates a survey invitation and builds the appropriate list: most panelists have never been invited to take a survey by AutoTrader.com before. The goal is to send out sufficient invitations to collect 350 completes per concept, to provide statistical validity.
2. Expose to Concept – Respondents answer a few profile questions, then read a neutral description of the concept. The next page goes to the actual concept and a sample landing page.
3. Assess the concept – Respondents then answer 11 standard questions about the concept, as well as a few that are specific to this particular concept.
4. Analyze results – AutoTrader.com’s research team cleans the data first, looking for speeders and streamliners. Then an analyst calculates an overall concept index, composed of an interest index and an execution index, which provides a convenient summary of the results. In addition, the analyst publishes the detailed results, using a standard report template.

Now, back to the title: “Cost Effective, In-House Normative Concept Test Program”.

• Cost-Effective – With Vovici as the only outside cost, the program was quite affordable. To run concept tests with AutoTrader.com’s external research vendors would have cost approximately $145,000 a year, but by bringing the research in-house, licensing Vovici, setting up standard templates, and relying on their own expertise, AutoTrader.com is able to perform the same work for approximately $20,000 a year. In fact, concept tests now take only six hours of labor per concept, resulting in annual savings of $125,000 a year. A typical large expense for outsourced projects is renting the sample, but AutoTrader.com saves this cost by using its house list.
• In-House – “We had always outsourced 100% of our research, so this was a brand new frontier for us. We wanted the right people on staff to program surveys and do the appropriate analyses,” Morgan said. “Lucky for us we had a couple of folks who came from the supplier side and were exposed to this type of work in the past.”
• Normative - How does a concept fare as compared to the other concepts that AutoTrader has tested in the past? Each questionnaire begins with the same 11 standard questions, some of which are used to calculate the interest index and the execution index. The interest index is a composite of their general interest, intent to purchase, and other aspects of product launch. The execution index includes the visual appeal, organization, ease of finding information and the user’s ability to reach their goal.
• Concept Test Program – To date, AutoTrader has tested 31 designs and adopted 50% of concepts as is, 25% after modification, while rejecting 25%. “Even if the rejection was 90% it would be okay,” Morgan explained, “because it would save the company money from developing unsuccessful products.”

The highest scoring concept tested to date is the AutoTrader.com Trade-In Marketplace, which just rolled out nationally, providing “a new and easier way to trade-in or sell your car.” AutoTrader.com calculates the trade-in value of the user’s car, which will then be honored by any of AutoTrader.com’s trade-in or buying-center partners.

One of Morgan’s internal customers, Kevin Gallagher, the senior director of product management, says, “The Concept Lab provides a quick and effective way to gather quantitative data earlier in the design process to help shape and refine innovative new product ideas. We’ve also had great success pairing the concept lab test approach with our more familiar usability and live site testing. The Concept Lab has been a great addition to AutoTrader.com.”

To see a recording of last week’s webinar, click this link.