Learn, unlearn and relearn – Adaptive Predictive Modelling during Pandemic

The sudden and dramatic impact of the COVID-19 pandemic has negatively affected the accuracy of many predictive models. In machine learning, unexpected changes in underlying data distribution over time is referred to as Model Drift. Almost each and every Data scientist working on scorecards and Predictive models would either have already faced or going to face the problem of Model drift due to current COVID-19 situation since models built to predict patterns, outcomes and behaviours are no longer viable.

All predictive models and scorecards essentially depend on the data being used for training it. The traditional supervised learning assumes that the training and the application data come from the same distribution. A Data shift its’ pattern due to many reasons such as consumer preferences, technological innovations, catastrophic events, etc. Catastrophic events like COVID-19 has brought a forced change in human behaviour and economic activities resulting from social distancing, self-isolation, lockdown, and other responses to the pandemic.

For example, consider the Banking industry, the after effect of this pandemic is bound to impact customer behaviour and the patterns of loan default or EMI payments. This drift is a phenomenon that creeps into Scorecards as time goes by and if not detected and treated on time, it can have detrimental effects on model’s performance.

Model Drift can happen because of following reasons:

1.      Distribution shift in the independent variables in train and test sets called as Covariate shift or Data drift. The primary reason behind the occurrence of the Covariate shift is Sample selection bias and non-stationary environments.

2.      Proportion shift in the target variable also known as Target Drift or prior probabilities shift

3.      Shift in the relationship between the independent and the target variable also called as Concept Drift

These Drifts can be gradual, incremental, cyclical or abrupt. For the catastrophic events appearing once in a decade or two, it shows sudden blips for few months before going back to normalcy.

 In the unparalleled scenario of COVID-19, if models are trained over the past year of customer behaviour data, the results would be puzzling because model has some normal data, some total lockdown data and some getting out of lockdown data. Therefore, adapting to the effect of COVID-19 inflicted Model Drift is the need of the hour. Delay in taking necessary actions might end up having serious consequences for the businesses.

Other challenges could be lack of relevant data. While the Spanish flu's impact is more like COVID-19, there is little data available about the 1918 pandemic. While past financial crises may help inform current economic models, the 2001 dot-com crash and 2008 financial crisis stemmed from entirely different circumstances.

Drifts can show following characteristics:

1.      Shift in distribution of target where model may show a positive bias in out-time validation since the majority of population shifted to bad class

2.      In case of mild drift, model may still be valid, but may only partially work.

3.      In the extreme scenario, severe drift occurs when maximum examples are misclassified. Misclassifications can be due to new pattern in data as well as changes in data and target distribution

Since drifts involve a statistical change in the data, the best approach to detect them is by monitoring its statistical properties, the model’s predictions, and their correlation with other factors i.e. if the model performance declines below some expected level, revaluate the model. It is important to consider whether changes in model performance metrics are due to sample bias or whether perceived drift is due to randomness or outliers, and not because of a shift in the data distribution or target. Another way to use the adaptive test statistics for drift detection such as Kolmogorov–Smirnov (KS) test indicates that the distribution of your prediction or residual values has changed.

One can also detect Model drift using an algorithm known as adaptive windowing that detects data drift over a stream of data. Adaptive windowing works by keeping track of several statistical properties of data within an adaptive window that automatically grows and shrinks.

In order to treat drift, one needs to work on data as well as adaptive modelling techniques. If we diagnose a concept drift, we must use judgment and domain knowledge to re-label the affected old data. If we diagnose a data drift, majority of the new data needs to be labelled to introduce new patterns. We might need to train the model in combination of old and new data when we find that both data and concept have drifted

Adaptive models can be implemented either through trigger based or evolving based approaches. Trigger based means that there is a signal which indicates a need for model change. The trigger directly influences how the new model should be constructed. The evolving methods on the contrary do not maintain an explicit link between the data progress and model construction and usually do not detect changes. They aim to build the most accurate classifier either by maintaining the ensemble weights or prototyping mechanisms. Ideal approaches can be shown in below given chart:

An ideal drift handling system should be able to quickly adapt to concept drift, be robust to noise and recognize and treat significant drift in model performance. It is good practice to detect and measure drift and specify a threshold to distinguish between major and minor drift. Additionally, you could refresh a model’s weight by extending its training on alternate data.

Very importantly, A/B test duration should span enough time to enable the modelling of the natural variability of a market cycle. The benefits and costs of A/B testing during these times need to be cautiously evaluated and balanced.

Understanding and detecting drift is a significant task. Ultimately, selecting the best methods for detecting and responding to drift often requires intimate knowledge of data, model, and application. Essentially, data scientists can no longer rely on historical data alone to train models and then deploy them in real-world scenarios. The ripple effect of the pandemic shows us that we need to be more agile, adaptive and leverage better strategies for keeping deployed models responsive and making sure that they provide the value they were built to provide.