Understanding Concept Drift and Data Drift for Robust Machine Learning Models

Concept drift is a phenomenon wherein the statistical properties of the target variable (y), which the model aims to predict, undergo changes over time.

Data drift, often referred to as virtual drift, occurs when the statistical properties of the inputs change. In the presence of drift, models built on historical data become obsolete, necessitating a revision of model assumptions based on the current data. Figure 1 illustrates the distinctions between concept drift and virtual (data) drift.

At the business level, examples of drift manifest in various use cases. For instance:

• Wind Power Prediction: Concept drift arises when predicting electric power generated from wind, given the nonstationary nature of winds and weather, contrasting offline dataset-based models with online training models.
• Spam Detection: Email content and presentation are subject to constant changes (data drift), and user interests evolving over time introduce concept drift. Users may begin or cease considering certain emails as spam.

Concept drift changes can take different forms:

• Sudden: The abrupt transition between an old and a new concept, exemplified by the behavioral changes during the COVID-19 pandemic, such as lockdowns altering population behaviors globally.
• Incremental/Gradual: The shift between concepts occurs gradually over time as the new concept emerges and adapts. An example is the transition from summer to winter.
• Recurring/Seasonal: Changes recur after the first observed occurrence, like seasonal shifts in weather affecting consumer behavior, such as buying coats in colder months.

Figure 2 illustrates how model drift detection functions. Initially, the system collects model inputs and outputs, calculates statistics over a time window, and compares them with either the sample set statistics saved during training or data from an older time window. The monitoring system saves various feature statistics and calculates the drift level using metrics such as Kolmogorov–Smirnov test, Kullback–Leibler divergence, Jensen–Shannon divergence, Hellinger distance, Standard score (Z-score), Chi-squared test, and Total variance distance