How Text Mining Changes Survey Analysis
Posted by Jeffrey Henning on Tue, Jan 26, 2010
At the 2010 Clarabridge Customer Connections conference, Todd While of Clarabridge moderated a discussion on market research and text mining with John Georgesen, Ph.D., senior director at Convergys; Michael House, division vice president at Maritz; and Scott Evans, Ph.D. of Harris Interactive. One of the key themes was the impact that text analytics has on traditional analysis of verbatim responses to surveys.
Prior to text analytics, analysts could not read all the verbatim responses for surveys that had collected tens of thousands of responses. For such surveys, they would draw a random sample of the responses, read them to develop a coding sheet of common categories of responses, then code or categorize each sampled response to develop some quantitative estimates of the frequency of occurrence of these codes. For large coding sheets, sometimes budget or time constraints would limit the coder to only manually classifying the verbatim responses according to the most frequent codes; in effect, ignoring the "long tail" of categories on the coding sheet.
Text analytics decreases the cost of coding high volumes of comments and increases the quality of that analysis. As John pointed out, text mining now lets you categorize every single response, which was impossible before when tracking hundreds of thousands of satisfaction surveys a month. Further, you can now categorize the entire coding sheet, since the work is automated. You can now identify low frequency negative customer experiences, picking up on rare events that would have been lost if coding was done manually.
Scott discussed how text analytics empowers the analyst to develop richer classifications. With traditional manual coding, you might have created a linear coding sheet from the bottom up. With text analytics, you can build a hierarchical classification scheme from the top down, allowing you to classify important detailed calls to action. As John put it, done properly, text mining enables you to develop new, richer ways of understanding current customer experiences, which may contradict the cherished traditional viewpoint held by the organization.
Michael highlighted that no longer does the code list have to be static, but it can change over time while still providing important trending information. In the past, analysts were reluctant to update or modify coding sheets, at the risk of breaking trending information. Introducing a new coding sheet required an abandonment of past tracking. Now, implementing a new classification scheme allows you to re-analyze historical data with the new scheme at no added cost; the process simply runs on the past data, providing important trending information.
Scott gave an example of how text mining had transformed the understanding of one Harris client. From past manual efforts, this organization felt that value and quality were synonymous in the minds of their customers. The closed-end questions they used did not reveal any significant differences. By analyzing the verbatim responses, Harris discovered that that quality was very broad in nature, referring to the brand and the positioning rather than to specific products and interactions. Value, on the other hand, referred to very concrete issues such as price, product set, feature and functionality. For this client, value and quality were not the same and the text mining showed that, to customer minds, their organization was high quality but low value. It enabled them to better understand their perceived positioning in the market.
To sum up, text mining is decreasing the cost of analyzing large quantities of verbatim responses, increasing the depth of verbatim analysis, and providing greater flexibility for adapting to changes in the data.