The Dark Side of Predictive Analytics

Predictive modeling is great. By predicting future events and behaviors, companies can increase their competitive edge and managers can make better informed decisions. Yet when there is too much faith placed in these models, and these models are left unchecked, the result can be gravely unintended consequences.

How can machines running statistical models go bad?

Meet the Feedback Loop

A feedback loop in modeling is where the results of the model are somehow fed back into the model (sometimes intentionally, other times not). One simple example might be an ad placement model.

Imagine you built a model determining where on a page to place an ad based on the webpage visitor. When a visitor in group A sees an ad on the left margin, he clicks on it. This click is fed back into the model, meaning left margin placement will have more weight when selecting where to place the ad when another group A visitor comes to your page. 

This is good, and in this case - intentional. The model is constantly retraining itself using a feedback loop. 

When feedback loops go bad...

Gaming the system. 

Build a better mousetrap.. the mice get smarter.

Imagine a predictive model developed to determine entrance into a university. Let's say when you initially built the model, you discovered that students who took German in high school seemed to be better students overall. Now as we all know, correlation is not causation. Perhaps this was just a blip in your data set, or maybe it was just the language most commonly offered at the better high schools. The truth is, you don't actually know. 

How can this be a problem?

Competition to get into universities (especially highly sought after universities) is fierce to say the least. There are entire industries designed to help students get past the admissions process. These industries use any insider knowledge they can glean, and may even try reverse engineering the admissions algorithm. 

The result - a feedback loop

These advisers will learn that taking German greatly increases a student's chance of admission at this imaginary university. Soon they will be advising prospective students (and their parents) who otherwise would not have any chance of being accepted into your school, to sign up for German classes. Well now you have a bunch of students, who may no longer be the best fit, making their way past your model. 

While ad clicks or college admissions are one thing, policing and criminal sentencing algorithms run the risk of being much more harmful.

Left unchecked, the feedback loop of a predictive criminal activity model in any large city in the United States will almost always teach the computer to emulate the worst of human behavior - racism, sexism, and class discrimination. 

Since minority males from poor neighborhoods dis-proportionally make up our current prison population, any model that takes race, sex, and economic status into account will inevitably determine a 19 year old black male from a poor neighborhood is a criminal. We will have then violated the basic tenant of our justice system - presumption of innocence until proven guilty.

What to do?

Feedback loops can be tough to anticipate, so one method to guard against them is to retrain your model every once in a while. I even suggest retooling the model (removing some factors in an attempt to determine if a rogue factor - i.e. German class, is holding too much weight in your model). 

And always keep in mind that these models are just that - models. They are not fortune tellers. Their accuracy should constantly be criticized and methods questioned.