When Machine Learning Models Decay
Machine learning models are only as good as the data they are trained on. Whether the learning is supervised, unsupervised, or reinforced underlying shifts in the data can cause issues. Two areas of concern in particular are data and concept drifts.
What happens when the underlying data that the model is trained on is no longer relevant? A famous example was when Zillow was impacted to the tune of $500 million. Likelihood is there were issues in the data and underlying assumptions.
The data up to that point was likely trained on a housing market that hadn’t seen a decline in over a decade. What happens when those assumptions are no longer true?
That’s why data and concept drift need to be closely monitored to understand if the model needs to be retrained or adjusted. Otherwise it can lead to catastrophic mistakes for businesses.
