Rabbit Trails and Predictive Analytics

One of the challenges and fun aspects of using BI data to analyze your business is the discovery process involved.

One of the things I do is look at the data we collect about customer sentiment and behavior on the content of one of our major Web properties. On one of our help and how-to Web sites, which includes about 35k assets, we capture data for things like page views on assets, customer ratings of those assets, customer verbatims (free-form comments), as well as a number of other things like SERP placement on major search engines, click-through rates, error rates, freshness of the content, top intent coverage as well as something we call deflection rate, which is how successfully an asset deflects a support call when a customer in is in the process of filing a ticket.

The goal of my most recent analysis was to see if we can detect a signal from the content that might predict changes to our overall satisfaction ratings for the product. We can compare specific questions in our satisfaction survey like, “How would you rate your experience setting up smartphones using our product?” to a content asset called “Set up your mobile device”.

An often neglected step in research is answering the question, “What types of decisions do I hope to make based on this data?"

One of the biggest challenges in data analysis, especially in predictive modeling is identifying whether there is causation or correlation in the data. For example, just because drinking coffee gives you a buzz doesn’t mean all buzzes come from coffee, however if everyone reports getting a buzz right after drinking coffee, you can show causation. If you  set up an A/B test with a control group, you can even prove it. Another, often neglected step in research is answering the question, “What types of decisions do I hope to make based on this data?” Sure, it’s fun wallowing in data, drawing conclusions, proving hypotheses, but if after you spend all that time and those resources, if all you can do is say, “interesting", then it was a waste. Some action or new strategy needs to emerge, like an"Employee Coffee Buzz Maximization Strategy".  In the article The Four Traps of Predictive Analysis, James Taylor says "Companies that don’t understand the kinds of decisions they want to make will struggle to get a return on their use of predictive analytics."

 As we started looking at the data signals coming from our content, I instinctively looked at the verbatim data. Historically we have been able to identify trends in the comments that are valuable feedback not only for the content developers, but also for the product engineers. Comments like, “Why don’t you make it easier for me to change this setting?”  are a very specific micro-signal which are very actionable. But these are not really predictive, and not necessarily a mind-blowing discovery. We have been sharing this type of feedback with product designers for years.  But what about all the other rich  data? Can we model trends in ratings or page views or any of our other signals and see if they are a predictor of satisfaction? We gathered all the data for 22 different survey questions, mapped them to the associated content assets and charted the data for those measures. Despite our efforts, we could not detect a pattern. We then dove deep into two specific areas that looked promising because they had large swings in satisfaction, one for the better, one for the worse. Conclusions? Again, no pattern, and in fact we discovered that the large swings in satisfaction were likely attributed to a small sample size for those questions, effectively negating the statistical significance of data.

Its critical to recognize that when changes occur in two different systems simultaneously it is impossible to show a cause/effect relationship.

So we took a step back. What were we seeing? Why was there no pattern? For starters, we only had 2 years’ worth of quarterly satisfaction data because our product was fairly new. Ideally you have 30 data points to detect a reliable pattern. We didn’t have enough points along that trend line. Second we observed that we were comparing data sets in two distinct systems, which although they have a relationship to each other, and were presumably surveying the same customers, both had independent changes affecting each system. The content team making efforts to boost the SEO of the content through keyword analysis, net promotion, and a number of other efforts. They were also consistently improving the content, making it better based on customer feedback. And, the engineering team was making incremental improvements to the product itself. Its critical to recognize that when changes occur in two different systems simultaneously it is impossible to show a cause/effect relationship of one system to the other. We had to reevaluate what business decisions we thought we could make.

When you start digging you notice other things, trends, signals, things that may seem unrelated but warrant further digging.

We were at a dead end. But I kept thinking about that valuable micro-signal we were getting from our verbatim data. As with any data mining project, when you start digging you notice other things, trends, signals, things that may seem unrelated but warrant further digging. I generated a hypotheses. What if we could mine a signal from the free-form feedback by categorizing it into either positive or negative categories, and separating the content feedback from the product feedback? We could turn it into quantitative data and chart it out with the other numerical data. I assembled my team and tasked them with analyzing and categorizing 1,000 verbatim comments. The results were much more interesting. At least we knew we were comparing things more closely related – comments specifically about the product with survey questions about the product. I also decided to add in support call volume data for the same areas and time periods and we noticed a correlation with decreased support call volume and increased satisfaction. Not earth shattering, but nice to validate.  One of our big “ah-ha” moments was when we looked at a chart combining the call volume data with the categorized verbatim data. At last, we were beginning to see the genesis of a signal. The correlative relationship was more evident and pronounced.

One of the limiting factors to to scaling this new discovery is that human categorization of free-form customer comments is costly, and time consuming. So, I decided to reach out to partners in the company working on machine learning and categorization of free-form data to see how we can apply those tools to the millions of strings of customer verbatims we collect. This work is ongoing and I can’t really share the results of this study. But some of the conclusions and learnings are universal. The data analysis process is iterative, and some learnings are unexpected. But sometimes dead ends and rabbit trails can lead to new discoveries.