Sentiment & Text Analysis: Man, Robot or Cyborg?
Posted by Jeffrey Henning on Sat, Jun 05, 2010
One of the most talked about blog posts this week was the Fresh Networks post by Matt Rhodes, The problem with automated sentiment analysis. Sentiment analysis typically tries to determine whether a statement is positive, negative or neutral. While the vendors often tout accuracy figures of 75%, the bulk of correct assessments are of neutral comments. Matt contends that because most comments are neutral, this overstates the accuracy, which fails at the type of comments most important to brands: positive and negative statements. Comparing machine coding of positive/negative comments to that of humans, accuracy ranged from 7% to 48% across the seven systems analyzed. For accuracy, Matt says that researchers need to code the sentiment.
In comments to the Research magazine recap of this research (Automated sentiment analysis gives poor showing in accuracy test), Mike Daniels says, "This study supports what is now generally considered a settled view - that automated analysis tools cannot, and generally do not pretend to, deliver the same levels of sentiment accuracy as well trained, fully briefed human analysts."
Where sentiment analysis looks to code comments as positive, negative or neutral, text analytics classifies comments according to lists or hierarchies of custom coding categories, also known as taxonomies or constructs.
A Conversition post, How to choose between human coders and automated coders, provides five guidelines for choosing between the methods. If you have a large sample size, a large number of constructs (coding categories) and a need for rapid turnaround, look at automation. If you have a small, static set of constructs and a proven team of reliable coders, stick with human analysis.
At the 15th annual CASRO Technology Conference this week, Jennifer Drolet and Chris Tonay of iModerate discussed their efforts to apply text analytics to qualitative analysis. The tools were inefficient and inaccurate due to small sample sizes (average of 60 responses) and the semi-structured nature of their interviews. They've decided that text analysis is at best a supplement to manual analysis; iModerate will use text analytics more for longitudinal studies and will pursue a blended approach of machine automation and manual methods.
As iModerate indicates, text analysis does not have to be an either/or decision - you can use text analytics (the robot) and then have a human review the decisions and correct them. This cyborg approach is in effect the state-of-the-art in translation software: CAT (Computer-Assisted Translation) has the software do the grunt work that it can do with high accuracy and then the human user does the work requiring brain power. The robots aren't winning yet.
Also see my post: Types of Surveys Suited for Text Analytics.