Machine Learning - the biggest thing since the compiler.
Over the last five or so years, there has been a revolution taking place in the world of software development.
"Machine Learning is the most significant thing to happen in the world of development since the compiler." - Mark Agar
Machine Learning is being mis-represented at the moment as being a tool for data scientist to gather ever deeper insights into data. It can do this, but that is just the tip of what it can do. In order to understand why Machine Learning is so important, you have to consider how it solves problems, and the types of problems that it is extraordinarily good at solving.
So, why do I think it is such an extraordinary shift in the way we develop software? Consider the following:
Give a developer, or technical architect a problem to solve. They will run to a whiteboard to labor over the detail, chat for hours about edge cases, and discuss for days how long it would take to develop and maintain and update and deploy. Only to tell you that most of what you ask for is impossible, and the rest will take about four man-years to get working… And, often, that would be completely fair and appropriate behavior!
Software development, until this point has relied on understanding the HOW to solve problems. If you could not understand HOW to solve the problem, you could not CODE it. And even if you did, it still could be very hard.
(Credit goes to Francois Chollet, the creator of Keras, for the simplicity of illustration in Deep Learning for Python)
The paradigm for Machine Learning is completely opposite to conventional development. Instead of having to worry about the endless, tiresome, detail of problems and problem spaces, you now find a set of examples of the problem being solved, and allow the machine to ‘learn’ how to solve it, resulting in a solution called a model which can then be applied to new examples to predict outcomes. Never once, did we trouble ourselves with the HOW, and never once did a person have to solve the problem.
This is a monumental shift!
And, even more remarkable, is that problems that were almost impossible to articulate, let alone solve are now, well, easy(ish)…
Image recognition, literature generation, text analysis, voice interaction, language translation, emotion detection, illness detection, crime prediction, weather modeling, hammering humans at chess, have all been proven to be utterly achievable in ways that until recently have been thought to be the things of science fiction.
If anybody needs a more compelling example of how Machine Learning changes the game, it took 4 hours for Google’s Alphazero to become the best playing chess entity in the world. https://en.wikipedia.org/wiki/AlphaZero
Let me say that again.
From scratch, Alphazero taught itself to play chess better than anything else in the world has ever been able to play it. And it did it in four hours (on admittedly pretty spectacular hardware). And what is more it doesn’t play like a human plays, https://www.technologyreview.com/s/609736/alpha-zeros-alien-chess-shows-the-power-and-the-peculiarity-of-ai/ .
The behemoths of Amazon, Google, Microsoft and Facebook have created enough momentum in this field to prove this is not a fad. There are three key reasons why the time is now good:
- Access to machine learning knowledge and software. Tensorflow, Theano, CNTK, AWS, Azure have all opened up the possibilities of advanced, mature frameworks, without huge R&D budgets or links to academic institutions.
- Processing power. Processing power is ridiculously cheap, and most PCs and Mac have enough grunt to build models, train, and test. And if you want to do something really big, AWS and Azure on hand to provide the processing power needed.
- Data. Boy do we have data! Sensors and tracking happens in every part of our lives and it’s only a matter of time before we have National Big Data day.
With all these things now available and established, technologists suddenly have these nearly magical tools at their disposal.
Which begs the question, why aren’t we all using them all the time?
I say that assuming you aren’t. Most companies aren’t – they probably do something with image classification with some internal process. But they aren’t exactly transforming themselves.
Machine Learning is a paradigm shift, and it will still take time to realize how to use it. There are two important principles that I am using to apply to my own way of thinking, which I hope you find helpful:
1 - Machine Learning allows us to solve problems that were previously difficult to even articulate, they remove established boundaries. This means to in order to take advantage of the technology you need to rethink what you are trying to solve. Forget the conditioning of the last few years, all the existing limitations, 'best' practice, and compromises that always limited your scope for change. Re-explore it again, now knowing you have something akin to magic to solve it with.
2 – Machine Learning does not just accelerate human learning, it can find new ways to solve problems. Alphazero did not just learn to play like humans:
“It doesn’t play like a human, and it doesn’t play like a program, It plays in a third, almost alien, way.” - Hassabis said at the Neural Information Processing Systems (NIPS) conference in Long Beach.
Rather than just looking at the processes you have in place, and exploring Machine Learning as a way to remove people (which it almost certainly will do), elevate the question. Allow machines the room to invent the way they solve problems, not just replay what you know, only faster:
You should not replace like with like, your processes, tools and products should change to make the most of this new technology. And of course, as is the way with the spinning jenny, those who resist, have the most to lose.
Over the coming posts I will be exploring how Machine Learning can be used to solve problems, invent products and transform software development, specifically in the world of digital marketing.
I am going to start with the world of testing, and how Machine Learning is a real threat to established ways of approaching that. The goal is to show how, when applied tactically, Machine Learning offers some advantages to how we work. However, if we're brave enough to rethink what we are doing, and play to its strengths it may well change the way we work for good.
I have no idea what I will conclude or where I will head after that, but I am eager to hear from others on a similar journey.
Photo caption:"This machine won't be replacing anyone anymore!" From http://www.environmentandsociety.org/mml/spinning-jenny-milena-bassen by Milena Bassen / CC BY
So well said and excited to partner with you where possible in the journey.
I'll buy you a pint to celebrate the first National Big Data Day.