Real-Time Reports Not Always Ready for Prime Time
Posted by Vovici Blog on Fri, Apr 08, 2011
Real-time reporting is a phenomenal feature of survey software. It’s fun to hit the Refresh button on the browser and see the additional answers roll in to a just-launched survey! Unfortunately, analysts need to keep in mind that such reports are works in progress. Trends in such data are really the illusions of trends – errors due to small sample sizes.
The following chart shows three random simulations of the first 300 answers from an eventual 9000 answers to a question rating a product on a 1-10 scale. In the first 100 responses, each series is following a different path to the eventual mean of 4.5. Even at a relatively large sample size of 300, the three series have means ranging from 4.3 to 4.7.
Analysts subject themselves to this on a monthly basis, when they provide a filtered view of survey responses for the month so far. The early averages are compared to past monthly averages and conclusions are reached, despite the fact that there is a wide margin of error for such early assessments.
Just because you can run a report in real time doesn’t mean you should jump to conclusions about that report.