Machine Learning Categories

I started reading a new book, “The Master Algorithm,” by Pedro Domingos, a professor at the University of Washington, and an artificial intelligence researcher.  I initially bought it as a tool to help me get very specific in my dealings with vendors, but I am actually quite enjoying the read beyond just brushing up.  In my house, I still have a dozen or more years of ACM and IEEE journals on things like Pattern Analysis and Machine Intelligence, Neural Networks, and Knowledge and Data Engineering, but my knowledge is a bit dated.

In my current position, I see loads of vendors and new products in the information security arena talking about machine learning as if it is some form of magical, mystical medicine that will magically solve all sorts of  security-related ailments.  What most of the vendors don’t realize is that I am fairly cognizant of these areas, and the generic answers about their “secret sauce” don’t impress me at all.  Once I figure out what school of thought their solution comes from, I start asking them how their solution might solve classes of problems that school normally cannot handle.  THAT is what differentiates a decent product from a mediocre one.

For me, some of the key questions in machine learning are:  

• How do we learn?  
• Can we trust what we have learned?  
• What can we predict?

Hundreds of new learning algorithms are invented every year by faculty and PhD/MS students, but they’re all derived from the five schools of thought which have been around for many years.  Each has its own strengths and weaknesses, and has classes of problems it can solve, and others it cannot.  The five schools, in the order I actually understand them best, are (in simple terms):

1. Connectionists, who try to emulate the brain and neuroscience, and use neural networks, and usually back-propagation, to solve their problems.  This was the focus of my MS degree back in the 1980’s.
2. Bayesians, who use statistics and probabilistic inference to solve their learning problems, using Bayes’ Theorem.  With an undergraduate degree in mathematics, I learned how this works much easier than most, but also understand its limited applicability.
3. Evolutionaries, who emulate evolution using ideas from genetics to search large spaces, and use genetic programming to evolve programs.  Work I did here was primarily in computational biology about 15 years ago.
4. Symbolists, who use logic and inverse deduction to attack their learning problems.  I always liked the elements of veracity that could be shown, along with solid evidence of reasoning ability from known facts to conclusions.
5. Analogizers, who learn by extrapolating from similarity between objects, and is the root of many scientific discoveries.  We make judgements of something based on its similarity with known things.  

Domingos’ book goes through each of these schools, with a main goal of trying to get more people thinking about how to unify these five schools of thought to eventually build “The Master Algorithm.”  

It has me thinking again, and I can only hope leads all the peddlers of “machine learning” and “big data” out there on the same journey.