How do you explain the data

In the last post I mentioned about Occam’s razor which talks about the case for not over-complicating things. Like anything else, there is a flip side to it – i.e. not over-simplifying things.  And some times as a result you run a risk of  showing a very dichotomous view of the big picture which doesn’t seem to add up to the details underneath it.  And, here is an example.

NAEP (National Assessment of Educational Progress), the only nationally-representative exam that measures student learning over the past few decades released a compilation of math scores for all 17-year-olds in the US. The goal is to measure what level of progress teaching methods have delivered in math learning abilities for students. The visual results of the study indicate that the performance hardly changed between 1992 and 2012. In fact, the average score dipped by a point (see the navy blue line below):

While this is not an encouraging finding, what is more baffling, as you notice in the chart is that none of the groups have a declining performance. In fact, every ethnic group has an upward trend in score performance in the ten year period.   This is called Simpson’s paradox, named after British Statistician Edward Simpson who described the phenomenon in 1951.

So what is the explanation?  This is where looking at the details of the composition of each group is important. As seen in the below table the composition of each group significantly changed between the two periods.

The modest progress made by each ethnic group is not visible in the overall results, because Hispanic and African-American student population have lower average scores.

Clearly, relying on single score measures such as averages or medians can be misleading in a situation where sub-groups change over time. While, this is a very important aspect in analysis of trends in public policy, economic policy and other research areas, there are areas where this can surface when analyzing trends in enterprise business analysis with customer groups or segments.

Explaining KPI trends

When you are tracking KPI’s across groups and rolling them up for a time-based trend/analysis, look out for the changing size effect of underlying groups. In some cases look for 'migration' of entities that belong to the group. Evolving definition of customer habits are a great example. Consumption changes or a certain demographic change can move an individual customer from what was previously a group X to a group Y. So in this case, whether you are trying to analyze for customer acquisition, behavior or conversion look for both effects to make sure your details align with the big picture you are trying to understand and convey.

If you enjoyed reading this post, please click the “thumbs up” icon above to  let me know your feedback! To see upcoming posts from this series in your LinkedIn news feed, click the "Follow" button at the top right of the page.

Credits: NAEP study from Brookings Institution.