Software Development as Plumbing

O'Reilly reminds us that software development has a lot in common with plumbing. Indeed, when my team worked on Bing Search for Xbox Entertainment, "plumbing" the data pipelines was a lot more challenging and time consuming than the ML part of the application

Here's the O'Reilly Newsletter by Mike Loukides sharing his opinion about it: 

We have to rethink the role of the programmer

Look for the industry to become more stratified and specialized. The programming world will increasingly be split between highly trained professionals and people who don’t have a deep background but have a lot of experience building things. The former group builds tools, frameworks, languages, and platforms; the latter group connects things and builds websites, mobile apps, and the like. These two types of programmers have always existed, mixing fluidly. We just haven’t recognized the distinction, and that’s going to change. A good analogy is plumbing. If you need to install a toilet, you call a plumber: they know how to connect things together. There are jobs for people who design plumbing fixtures, but you wouldn’t want them working in your bathroom.

We need to think about how programming is taught

Like reading, some people learn how to code with little training, and others don’t. But as with reading, we shouldn’t accept a world in which some people enter primary school programming-literate, and those that don’t have to wait until high school. We’ll need teachers who are trained in teaching programming—specifically, teaching programming in the early grades. We already have programming environments that are optimized for teaching children, including Scratch, Alice, and their relatives. And don’t discount the role gaming could play. Minecraft has unwittingly taught a generation of grade-schoolers how to program in Java.

We also need to build bridges for people with great programming skills but without a deep computer science background—the plumbers—to enter the professional market. Some of those bridges exist already; they include the many boot camps and schools like General Assembly and Holberton. These are distinct from college degree-granting programs (the traditional computer science major) and serve a different purpose. They’re more like vocational education programs: They’re focused on practice, with minimal emphasis on theory. They’re about learning to program in a professional context—working with a web platform, a database, or even an AI platform—but not about developing those platforms or databases. They’re for those who say, “Why should I know how to program quicksort? If I want to sort something, I’ll call a library function.” That’s just fine, and we shouldn’t pretend that it isn’t.

In contrast, CS majors should continue to be exposed to and work with theory and algorithms—not because they’re going to write their own quicksort but because we need people who can develop and implement new algorithms, and the best way to learn is to practice on algorithms we already understand. You don’t need to be good at math to program, but you do need math to push computing forward—particularly if you’re interested in data science or artificial intelligence.

We need new, more sophisticated programming tools

In “Hidden Technical Debt in Machine Learning Systems,” the authors—a group of researchers and engineers from Google—argue that machine learning is a relatively small part of any application. Much of the rest is wiring things together: building data pipelines, connecting the application to the serving infrastructure, providing for monitoring. It’s not glamorous, but it needs to be done, and done correctly. I’d bet that much more downtime results from bad plumbing than from bad implementations of ML algorithms. Rather than relying on our current crop of languages, I wonder whether or not there are better languages for this part of the enterprise. It’s long seemed strange to me that programming languages aren’t all that different from what they were in the 1960s and 1970s: line-oriented, alphanumeric texts, most often in fixed-width type. Functional languages date back to the 1950s, and the earliest roots of object-oriented programming aren’t much later. What would it mean to imagine other kinds of languages?