Assume the Worst of that Survey
Posted by Vovici Blog on Thu, Mar 10, 2011
“According to the survey, 20% of consumers plan to move in the next year.” What, if anything, does that mean?
It can be hard to tell how seriously to take the results of a survey, since they vary tremendously in scope and representativity. I’ve done quick-and-dirty surveys of five people in an afternoon (for instance, calling up industry analysts and asking them two or three questions) to surveys that costs hundreds of thousands of dollars and were carefully constructed to project to the United States. At the high end, there are federal surveys that cost hundreds of millions of dollars to conduct.
However, whenever you read the results of a survey, and no details for how it was collected are given, I want you to assume the worst.
- Assume the survey has a small sample size. If the sample size isn’t provided, assume only a few dozen people were interviewed. I like to calculate the lowest sample possible for the range of percentages shown – for instance, one chart I saw at a recent conference showed that every choice had a nice even multiple of 5%. I presumed 20 people were interviewed.
- Assume it’s old and unreflective of conditions today. If no date is given, assume the survey is out of date – maybe years out of date. I recently tracked down a survey a nominal competitor of ours had written about (without providing the source), only to find out it was over five years old and from an analyst firm that no longer wanted its name associated with it (precisely because it was old).
- Assume it’s not representative of anyone other than those surveyed. If there’s no discussion of how respondents were chosen, assume it is a convenience sample. A firm that took the time to create and use a random sample that is projectable would take the time to tell you that they did so, because they’ve spent a lot more money on their survey than is typical.
- Assume it’s meaningless. Many surveys are factoids – “something that looks like a fact, could be a fact, but in fact is not a fact,” as Wesley Pruden put it. If you don’t have any context about that survey, it may mean something quite different than you think it does.
Caveat lector – let the reader beware.
Am I being hard on surveys? Not at all. Adam Curtis, writing in the February issue of Research magazine, gives many examples of advertising claims, for instance, based on samples of 40 to 70 respondents. And, remember, large sample sizes are no guarantee of accuracy – or perhaps John McCain would be President (based on a 2008 survey of 270,000 respondents); random sampling trumps convenience sampling, every time.
Now if you don’t want your colleagues to assume the worst of that survey you just conducted, provide them the background on when and how you collected the results. If you used a convenience sample, discuss how it may or may not be representative of your target audience. AAPOR members who publish survey results are required to disclose much more detail about how a survey was conducted (see Standards for Minimal Disclosure). Look to them for inspiration on what facts to provide.
So, back to the original statement, “According to the survey, 20% of consumers plan to move in the next year.” What does it mean? I’m assuming it means absolutely nothing.