Which Machine Learning - Deep Learning Technique to choose? Presenting a case for LightGBM

When it comes to Machine Learning Algorithms our thoughts first go to Neural Networks and then to bagging and Boosting algorithms. Among the Boosting algorithms, a lot of work has been done using Gradient Boosting and recently XgBoost is also being used a lot. But if it comes to choosing one algorithm as the latest favorite then the foremost candidate if LightGBM. Given its recent success at getting the best results on various Kaggle competitions; it seems to be a weapon of choice for most Data Scientists.

               This is not to say that it’s the algorithm which defines the end result. Data is and will remain the key factor for deciding the model strength followed by the parameter tuning by an expert. Algorithms need to be selected for their correct fitment and results.

               Neural networks are an extensive family of algorithms which have dedicated usage. For example, CNN is primarily being used for image processing which LSTM is being used for time based decisioning where repeat instances for the same entity are available. Again, Neural networks are demanding and need careful parameter tuning and multiple iterations to really stabilize and provide good results.

               Random Forests, on the other hand are fast and easy to develop but suffer from over fitting and in practice they are generally out performed by boosting models.

               Gradient Boosting models - GBM are very successful for classification problems and given that the sum of multiple sequential models is better than multiple strong parallel models; GBM generally gives better results than Random Forests.

XgBoost employs two key differentiators to GBM. First, It uses L1 & L2 regularization (penalization of estimator coefficients for error reduction). Second, it also enables the model development with parallel processing and hence can handle large datasets with ease.

               However, the trophy goes to LightGBM as it develops unbalanced trees based on loss reduction in a particular leaf and does not limit itself to populating all the nodes at the same level. Hence LightGBM can go to more depth in the data and extract information which would otherwise be missed by GBM & XgBoost as they max out on the depth. Also LightGBM allows the jumping of levels for cumulative loss reduction, I.e., if at level 3 the loss reduction is 0 but further on at level 4 split the loss reduction is -3 then overall there is more loss reduction till level 4. Other boosting algorithms generally will stop at level 3 where they hit 0 loss reduction. However, LightGBM will continue to search for further splits as long as the cumulative loss reduction is there. This allows LightGBM to develop very detailed data maps for predictions. Needless to say this is a problem too as it can easily end up in over fitting but with right expertise, LightGBM ends up as the best predictor algorithms.

               So let’s gear up and start exploring LightGBM more and more.

References:

1-     https://lightgbm.readthedocs.io/en/latest/

2-     https://github.com/Microsoft/LightGBM

3-     https://papers.nips.cc/paper/6907-lightgbm-a-highly-efficient-gradient-boosting-decision-tree.pdf