Will Machine Learning Revive The Generalist?

The alumni association of the George Washington University (one of my alma maters) has a virtual book club in which I occasionally participate. The most recent selection we read was Range: Why Generalists Triumph in a Specialized World by David Epstein. In it he sets out “to explore how to capture and cultivate the power of breadth, diverse experience, and interdisciplinary exploration” in a world that “increasingly demands hyperspecialization.” He concludes that “a diverse group of specialists cannot fully replace the contributions of broad individuals.”

Previously, in Part 1, I wrote about Gary Kasparov and his insights into artificial intelligence and machine learning he shared in a Wired magazine interview. While not quite using these terms, he differentiated between the deep learning of an AI machine and the broad range of human understanding “… which is clearly a sign of human intelligence.” He noted that people can take knowledge gained in one realm and apply it elsewhere. “Let's say you accumulated knowledge in one game. Can [a machine] transfer this knowledge to another game, which might be similar but not the same? Humans can. With computers, in most cases you have to start from scratch.”

Indeed, after IBM’s Watson established complete domination over two past champions during a game of Jeopardy! IBM turned to medicine. Dr. Watson would be a brilliant AI doctor eliminating diagnosis errors, optimizing treatments, and even alleviating doctor shortages. However, after 9 years, he is more like an AI assistant performing certain routine tasks well but struggling to make sense of complex medical information. The training that Watson received that enabled it to be successful at Jeopardy! Did not readily transfer to other problems.

 Epstein in Range also picked up on Kasparov’s insight. He quotes Kasparov: “anything we can do, and we know how to do it, machines will do better.” However, Kasparov also noted that machines and humans frequently have opposite strengths and weaknesses. He went on to organize “advance chess” tournaments where machines are paired with humans. Machines take care of the tactics, which is what they are good at, and humans determine strategy, which is what they are good at. To win in those tournaments, he declares, “human creativity was even more paramount … not less.” This is an example of what is known to AI researchers as Moravec’s paradox: machines find the difficult things easy and the easy things difficult. It is relatively easy to write programs that use logic, solve math problems and play mental games like chess, Go and even Jeopardy! It is much harder to write a program that can conceive of a strategy, create new goals and objectives and use commonsense reasoning.

Leonardo da Vinci (as Walter Isaacson noted in his book by that name) had an “instinct for discerning patterns across disciplines...” Many of Leonardo’s amazing achievements and discoveries were the result of his transferring what he learned in in one discipline and applying it to another. Leonardo was able to see patterns in nature, and he theorized by making analogies instead of using mathematical tools like Newton and Galileo. He saw similar patterns with the branching in trees, human arteries and rivers or the way light, sound, and magnetism spread in radiating waves. “With his keen observational skills across multiple disciplines, he discerned recurring patterns,” Isaacson writes. (Leonardo was also less than diligent in publishing his findings; many treatises were started but never published or even finished. Hence, many of his discoveries had to be rediscovered by others, sometimes centuries later. But that’s a topic for another article.)

In contract, the medieval guild system in Europe, which was still flourishing in Leonardo’s time, sought to maintain and protect specialized skills and trades. While this resulted in highly trained and specialized individuals who perfected their trade through prolonged apprenticeships, it also encouraged conservatism and stifled innovation. It ushered in an era of stagnation that did not end until the Renaissance.

Over the past few years, I’ve seen companies and society again move away from valuing generalists. Within the airline industry, in the immediate post-deregulation period starting in 1978, a success required a broad knowledge of the industry and its many facets: flight scheduling, crew scheduling, airport staffing and operations, flight operations, marketing, pricing, yield or revenue management, and many more. Once the disruptions and challenges of deregulation faded, to the fore came the specialist who spent an entire career in one department with a deep, deep understanding of its role and function, but with little appreciation of or insight into other departments. This is not unique to airlines. As industries and the companies in them grow and mature, Epstein observes, “vertical-thinking hyperspecialists would continue to be valued but lateral-thinking generalists would not.”

That is not necessarily a good thing or healthy for a company. In a study by Northwestern and Stanford, researchers found that “in professional networks that act as fertile soil for successful groups, individuals moved easily among teams, crossing organizational and disciplinary boundaries and finding new collaborators,” while unsuccessful ones were “broken into small, isolated clusters in which the same people collaborated over and over.” Epstein cites Arturo Casadevall, an eminent microbiologist at John Hopkins, who worries that “we fill people up with facts from courses,” while “what is needed is just some background, and then the tools for thinking.”

We may be at an era where the role of human specialists within a high performing team will be replaced, at least in part, by machines. Like Kasparov, we will partner with a machine in a real-life equivalent of “advance chess”. With machine learning systems leveraging increasingly powerful and specialized processors, computers will deliver deep expert knowledge. They will solve the logic and math problems, uncover patterns in data and all the other things that humans are not good at. In turn, people will collaborate within and between teams, determine strategy and goals and innovate. Human creativity will be again be paramount and the generalist who form relationships and can see analogies across departments and functions will once again become highly valued by companies. They will train and guide the machines to solve the problems that need solving. To paraphrase the subtitle to Epstein’s book, human generalists will triumph in a specialized machine world.

I leave you with the words of Leo Tolstoy, from his classic War and Peace: “And he refused to specialize in anything, preferring to keep an eye on the overall estate rather than any of its parts…And Nikolay’s management produced the most beautiful results.”