Recommended Sample Size for Accurate Surveys
Posted by Jeffrey Henning on Fri, Feb 27, 2009
I'm often asked, "How many responses do I need to get to my survey for it to be accurate?"
First, if you are posting a link to your survey on blogs and Twitter feeds, it will not be representative of any target population, and no number of responses is going to make it so. The information will be interesting from a qualitative standpoint, but-since it is not a random sample-it is not quantitative. In this case, whether you get one hundred, one thousand or one million responses doesn't matter. The information is interesting to talk about and might be fine for illustrative purposes for a blog post or a webinar, but the findings will not be useful for decision making. (For instance, see the convenience sample I used to survey list unsubscription methods. Good for a blog post, bad for a vendor estimating the market opportunity of a new unsubscription technology.)
If you are doing a random sample or attempting a census, then the statistical accuracy can be measured. The two key measures are confidence level and margin of error, and they work together. If you have a 95% confidence level and a 5% margin of error and a question choice selected by 50% of the sample, then 19 out of 20 times the true answer-the answer you would find if every single person in the target population answered this question-can be assumed to be in the range of 45% to 55%.
The accompanying table summarizes the number of responses you need, depending on the number of people in your target population (e.g., how many employees or customers you have).
Because response rates to surveys are not 100%, we advise you to try to conduct a census for target populations of 1,000 or less. Send out a survey invitation to everyone in the target group, followed by periodic reminders.
For populations greater than 1,000, invite a random selection of 1,000 possible participants to take the survey. This way the next survey that you conduct will not need to go out to all the same people, improving your response rates for future research.
Now, you may often wish to crosstabulate survey results across key subsegments. If it is very important that you have a 95% confidence level and 5% margin of error for leading subsegments, then you will want to invite a much larger group to take your survey. The Raosoft Sample Size Calculator can help you determine appropriate sizes. Weigh the importance of that increased accuracy versus the need for other surveys of your target group, and live with the larger margin of error for subgroups, if you can.