Welcome to the Listening Post!

The Road to Nowhere: Recognizing Respondents' Cognitive Limits

Tweet

 

Well, we know where we're going,
but we don't know where we've been.
And we know what we're knowing,
but we can't say what we've seen.
And we're not little children,
and we know what we want.
And the future is certain,
Give us time to work it out.

- Talking Heads, “Road to Nowhere”

There are five perfect albums in the history of mankind, albums on which every single song is a masterpiece. One of these is Little Creatures by the Talking Heads. (You can argue about the other four in the comments.)

I’m pretty sure that “Road to Nowhere”, on that album, is not in fact a David Byrne treatise on the perils of ignoring respondents’ cognitive limits. But perhaps its lyrics can offer a cautionary reminder to questionnaire writers.

Well, we know where we're going, but we don't know where we've been. Like most respondents, I cannot in fact accurately tell you how many times I have been grocery shopping at Stop and Shop in the past four weeks. I can estimate it, but it is simply an estimate and if you calibrate it with the store’s actual data, you will find that its customers are overestimating visits by as much as 50%. Questionnaire writers love questions about category usage and store visits but such self-assessments are not accurate; for rarely used categories or rarely visited stores, especially, respondents will naturally think of their last visit and will think it occurred more recently than it did. Don’t assume you can extrapolate from such questions to the overall market but do use such questions to differentiate more frequent users of a category from less frequent users.

And we know what we're knowing, but we can't say what we've seen. Keep in mind that questionnaires are not the only market research technique, and surveys can’t pull information out of respondents’ minds that they are not consciously aware of.

• Phil Barden of Decode Marketing, at the Online Research Methods conference in January, showed how brains exert more energy looking at logos for brands they are not aware of than they do for brands that are familiar with: not recognizing a brand is mentally harder than recognizing a brand. Now imagine the poor respondent, asking themselves, “Did I see that brand’s logo in a banner ad the past month or do I think I did because I am so familiar with the brand?”
• Many consumer decisions are made impulsively and rationalized after the fact; the reasons a respondent gives are not the reasons they bought.
• Finally, no respondent can describe everything they look at or choose to ignore when shopping the aisle of a store – eye tracking and neural monitoring within stores, as emSense does, provides a fuller picture of thousands of unspoken aspects of respondent behavior (capped off with a survey to supplement the data). Heat maps can reveal what actually caught the respondent’s eye, and neural monitoring can reveal their rational and emotional reaction to what they were looking at.

And we're not little children, and we know what we want. Many questionnaires would probably be better if they used questions that children could answer; too many surveys assume that respondents do in fact know what they want. This becomes especially difficult to evaluate when testing new product concepts. New products may test poorly in a focus group, because participants don’t have enough time to get past the novelty of a concept into how they could actually use it. Many surveys focus on the rational side of the respondent, not the emotional side. When I bought an iPad in the first week it came out, yes, I rationally wanted a device to surf the Web on from my couch; but I bought because emotionally I didn’t want to miss the next major platform shift, because of my self-identity as an early adopter. Would I have realized that when answering a purchase intent question? Probably not.

And the future is certain, give us time to work it out. Fundamentally market research can provide guidance to what the future of markets will look like, but market research cannot provide certainty. And survey questions that assume such certainty on the part of respondents do a disservice to respondent and analyst alike.

What you will find is that however bad a question you write respondents will try and answer it. And that really is a road to nowhere.