Do not take the easy path - a Machine Learning perspective

As many of you have seen, machine learning and its applications are virtually everywhere today. NO industry today runs without it, no company exists today that is, at the least, not worried about it. Whether this is a hype or not is a topic for another day. But it has been proven that machine learning has a wide range of applications with immense potential to affect both topline and bottomline of businesses as well as unearth completely new revenue streams and business models. The attention it commands, from board rooms to assembly lines, is not without reason.

A lot of you have no doubt seen many machine learning & AI related articles being shared, either on LinkedIn, or Facebook, or Flipboard, or Medium, or some other equivalent channel. Today, I want to draw your careful attention to a certain subset of those blogs & articles.

At its core, machine learning is not easy. It involves significant knowledge of linear algebra, statistics, probablity theory, stochastic processes, calculus, optimization, algorithms, programming, and this is without even going into issues of distributed and parallel high-performance computation. Naturally, not everyone understands one, or any, of these topics, but they want to know about what AI is all about. This has led to a proliferation of articles that promise to "teach" you machine learning, without having to go through the "horrors of the math" or "pains of programming". Essentially, these materials claim to make you AI-savvy without getting blood on your hands. Materials in this context are not just limited to blogs & articles but also tools, BI software, etc, that seek to abstract away the chaos of the kitchens, and only serve you up the prepared, delectable meal. Naturally, people LOVE it.

I have to admit, such resources do have some merit. It allows the democratization of an otherwise technical field to a much wider audience, lets people and firms make progress, and enthusiasts indulge in trivia. However, it is very important to keep the target audience in mind. And I strongly believe in this:

If you are, or intend to be, a student of machine learning, or a serious consumer (even at a CXO level) of data-driven products and solutions, do not fall for this easy bypass route.

While this might not exactly be music to your ears, true knowledge does not come from shortcuts. You cannot launch a Mars mission by reading popular science magazines. Similarly, you cannot appreciate the "how" and "why" of machine learning by choosing to steer clear of the "difficult" portions.

I'll give you a short example. Very often, when I interview candidates for a data scientist position, I notice their resumes proudly displaying their participation and completion in Andrew NGs famous Coursera course on Machine Learning. Before you scream blasphemy, I openly admit - its a great course, one of the best of its kind. But it too has a target audience. And the applicants in question are engineers. The first question I ask them is "Have you actually also gone through all the material in CS229?". This is the official, non-watered-down course that is publicly hosted by Stanford online. For an engineer and aspiring data scientist, I completely do not see why they would not do this version - which covers the actual math that Andrew (rightly) glosses over in the Coursera course - because it is meant for a different audience.

For the students: you NEED to know the theory as well as the practice (coding). No, really. No data scientist ever made it big without understanding probability distributions, or spending frustrated late hours debugging python and R codes. Nobody created wonders with neural nets without dragging themselves through non-convex optimization for backpropagation. DO NOT take the easy way out. It is a road that leads to a crowded pool of mediocracy, where there are few takers.

For the CXOs, the startup-kings and the product managers: while it may not be essential that you know how to code yourself, yes IT IS important that you understand what is going on, than view it as "black magic which that guy in the corner can do". You need to know the foundational concepts - curve fitting, cost functions, bias-variance tradeoffs, train & test splitting, model selection, and parameter tuning. You need to be able to ask whether the settings your scientist happily plugged in were established through cross-validation or mood swings. You need to be able to know whether a result is statistically significant, and what that means for your business.

Regardless of your background, the knowledge of what happens under-the-hood in machine learning is too precious to be left as a black box today, and you'll do your career and company a MAJOR service by educating yourself about it.