Welcome to the Listening Post!

Harry Potter and the Echo Chamber: Social Media Research Validity

Tweet

 

For the Global Online Moderator Community, Theresa Sorensen recently wrote a post about social media monitoring and Harry Potter:

Social Media Monitoring tools are not perfect and they do not, at the click of the button, provide you with the answer.  There is no magic way of knowing with the push of a button what insights are available for a brand or if that brand is positively or negatively received in social networks by social consumers…

Researchers have to bring their magic into the mix! Harry Potter will not be showing up and providing you with the answer with a wave of his wand.  A social media monitoring tool is just that - a tool.  It’s a tool to collect data.  As researchers, you must then apply your skills and expertise to making sense of that data, whether this is a qualitative, quantitative, or a mixed methodological approach.  The data is only just data without you!

Theresa makes a great point. Social media monitoring is not like probability sampling; you cannot simply extrapolate from your results and determine what the public thinks within a certain margin of error.

To reach quantifiably representative conclusions from social media market research, you need to develop mathematical models. And these models may need to be quite specific to the industry you are researching. So far, researchers have publicly discussed models to predict weekend box-office receipts, consumer confidence, presidential approval, and U.S. Congressional elections:

• Weekend box-office receipts – In the study “Predicting the Future with Social Media” (a title that would put Harry Potter’s Divination class to shame), Sitaram Asur and Bernardo A. Huberman of HP Labs demonstrate that “a simple model built from the rate at which tweets are created about particular topics can outperform market-based predictors”; in this case, for predicting the opening box office receipts of U.S. movies. Their model predicts such receipts with 97.3% accuracy, better than a prediction market. The authors also built a model using basic sentiment analysis to predict the second week of box office receipts, which are assumed to be a function of positive word of mouth from the first weekend.
• Consumer confidence – Brendan O’Connor, et al, in the study “From Tweets to Polls: Linking Text Sentiment to Public Opinion Time Series”, found a 73.1% correlation between the sentiment of tweets about “jobs” and the Gallup daily measurement of consumer confidence.
• Presidential approval – O’Connor also found a 72.5% correlation between sentiment in tweets about Obama and presidential job approval polls.
• Congressional elections – Every two years, in the U.S., all 435 members of the House of Representatives are up for election, as well as 33 or 34 Senators, making for almost 470 Congressional races to be tracked – a pollster’s nightmare. The Daily Beast and WiseWindow built a social media model that correctly predicted 97% of Senate races and 87% of the House races that it tracked (the 77 most competitive races).

The good news is that you can use social media monitoring, and even basic sentiment analysis, to build models with predictive validity. The bad news? Many of these applications are going to be limited to brands with a high volume of social media activity. Moreover, models may be fragile, given the rapid change in social networks and their expanding adoption. The models may show skewed results when influencers are retweeted and quoted across the social media echo chamber. And, finally, models may be scammed: for instance, to mislead their opponents, political candidates might inundate social media networks with astroturf comments (fake posts of “grass roots” support), skewing the results of such models.

The final good news for professional market researchers, though – instead of spending all your money on data collection, you’re going to spend most of it on model building. If you’re the analyst building the model, this is probably more enjoyable and will help you create an asset with ongoing value.

Though maybe some of your models will be shortlived. I reached out to Sitaram Asur to ask him if he had kept his Hollywood box office model active, but he hadn’t. Which is too bad, as I wanted to know his forecast for the opening U.S. box office receipts for “Harry Potter and the Deathly Hallows”. Regardless, social media market research gives you the opportunity to work your own magic.