Text Analysis: Man vs. Bot
Posted by Vovici Blog on Tue, May 10, 2011
This guest post is by Theo Downes-Le Guin, chief research officer for Market Strategies. Theo co-founded Doxus, which Market Strategies aquired; Doxus was a research consultancy providing marketing and product strategy to leading technology corporations.
Early in my market research career I did a lot of qualitative analysis. A typical project consisted of a handful of focus groups; my job was to review every video tape and transcript. We employed elaborate strategies of analysis including behavioral coding and cutting-and-pasting (literally) transcripts to compare responses across sessions.
Today most qualitative MR analysis is a hasty and impressionistic affair. The industry now has many more skilled practitioners who can turn out serviceable deliverables without the help of analysts or software. We can also rely more on source material (images, audio, video). But, mostly, expectations have changed: clients require qualitative reports almost immediately on conclusion of fieldwork, which sends a certain message about the role of measured textual enquiry.
As traditional qualitative MR has become less structured and disciplined in its analytic practices, the tools at our disposal for reliably handing qualitative data have become more sophisticated and usable than ever. These tools are critical to interpreting the massive quantities of text that confront MR from sources such as social media, call-center records, and survey verbatims. These forms of text now receive systematic and skilled treatment at the hands of trained analysts similar to what the text of focus groups received 20 years ago.
In an experiment last year conducted by Market Strategies International, we compared a traditional qualitative analysis process led by senior moderators (no transcripts) against a software-driven process led by a junior analyst (transcripts as sole source material). The experiment suggests that text analysis platforms aren’t just for unlocking the secrets of social media: they hold promise for a return to rigor for traditional qual as well, if we want it and are willing to pay for transcripts.
The two analyses (conducted in parallel but separately) achieved equal content validity: that is they both represented all aspects of the subject being researched. The analysis that used humans rather than text analysis software tended to weight emotional inputs more heavily. And the moderators did a better job recognizing ironic or contextual modifiers that required additional coding for the software to interpret correctly.
As well, the software tended to find shorter, simpler themes while the human analysis approach tended to synthesize multiple themes together, providing findings that might be sophisticated but also might be confusing. Of course, seasoned analysts can form, support and discard hypotheses quite fluidly. This can be both a blessing and a curse in the search for insights that are down-to-earth and yet also scientifically objective.
Personally I’d like to see a little more rigor returned to the analysis of in-person qual research, if only to ensure that social media analysis doesn’t eventually slide into impressionism as well. As we move into research modes in which text data are produced by listening rather than by asking, we need to find a way to cope with the fact that the role of the analyst can be diminished and transformed in counterproductive ways. Automated word clouds and sentiment indices, even when based on sophisticated algorithms, do not provide sufficient insight. We need more consistent discipline around how to integrate manual and software-based analysis for all modes of research that revolve around text.