Software engineers and software artisans

Let us look at the differences between the kind of activities we call "engineering" as opposed to artisanship or craftsmanship. It will then become apparent that today's computer programmers are software artisans rather than software engineers. Several important conclusions will follow.

Engineering disciplines

Consider what kinds of things a mechanical engineer, a chemical engineer, or an electrical engineer does, and what kind of studies they require for proficiency in their profession.

A mechanical engineer studies calculus, linear algebra, basics of differential geometry, and several areas of physics such as theoretical mechanics or elasticity theory, and then uses calculations to guide the design of a bridge, say. A chemical engineer studies chemistry, thermodynamics, calculus, geometry, some areas of physics such as thermodynamics and kinetic theory, and uses calculations to guide the design of a chemical process, say. An electrical engineer studies advanced calculus, as well as several areas of physics such as electrodynamics and quantum physics, and uses calculations to guide the design of an antenna or a microchip.

The pattern here is that an engineer uses science and mathematics in order to help build things. Mathematical calculations and scientific reasoning are required before anyone starts drawing a design, let alone building a real device or construction.

Some of the studies required for engineers include somewhat arcane mathematical concepts such as a "Lagrangian with non-holonomic constraints" (those kind of systems are common in robotics), the "Gibbs free energy" (for chemical reactor design), or the Fourier transform of the delta function and the inverse Z-transform (for digital signal processing).

To be sure, a large part of what engineers do will not be covered by any theory: the know-how, the informal reasoning, the traditional knowledge passed on from expert to novice, - all those skills that are hard to formalize. Nevertheless, engineering is crucially based on science and mathematics for some of its decision-making about new designs.

Artisanship: Trades and crafts

Now consider what kinds of things shoemakers, plumbers, or home painters do, and what they have to learn in order to become proficient in their profession.

All these trades operate entirely from tradition and practical experience. A novice shoemaker needs only to take interest in how leather is cut and put together, say, and to start trying it out in a home workshop. Apprenticeships proceed via learning by doing while listening to comments and instructions from an expert. After a few years of apprenticeship (for example, a painter apprenticeship in California can be as short as 2 years), a new specialist is ready to start productive work.

The trades do not require an academic study because there is no formal theory from which to proceed. To be sure, there is a lot to learn in the crafts, and it takes a large amount of effort to become a good artisan in any profession. But there is no rank-4 elasticity tensor to take into account, nor any differential equations to solve; no "Fourier transforms" to apply to "delta functions", and no "Lagrangians" to check for "non-holonomic constraints".

Artisans do not need to study any formal science or mathematics because their professions do not make use of any formal computation in order to guide their designs or processes.

Programmers today are artisans, not engineers

Now I will argue that programmers are not engineers in the sense we ordinarily understand the engineering professions.

No requirement of formal study

According to this recent Stack Overflow survey, about half of the programmers do not have a degree in Computer Science. I am one myself; my degrees are in physics, and I have never formally studied computer science. I took no academic courses in algorithms, data structures, computer networks, compilers, programming languages, or any other topics ordinarily included in the academic study of "computer science". None of the courses I took at university or at graduate school were geared towards programming. I am a completely self-taught software developer.

There is a large number of successful programmers who never studied at a college, or perhaps never studied formally in any sense. They acquired all their knowledge and skills through self-study and practical work. Robert C. Martin is one such prominent example; an outspoken guru in the arts of programming who has seen it all, he routinely refers to programmers as artisans and uses the appropriate imagery (novices, trade and craft, the "guild", etc.). He compares programmers with plumbers, electricians (but not electrical engineers!), lawyers, and surgeons (but not mathematicians, physicists, or chemists). According to one of his blog posts, he started working at age 17 as a self-taught programmer, and then went on to more jobs in the software industry; he never mentions going to college. It is clear that he did not need years of academic study in order to become an expert craftsman.

Here is another opinion:

Software Engineering is unique among the STEM careers in that it absolutely does  not require a college degree to be successful. It most certainly does not require licensing or certification. It requires experience.

This is a description that obviously fits a career in crafts - but certainly not a career, say, in mechanical engineering.

The high demand for software developers gave rise to "developer boot camps" - vocational schools that prepare new programmers very quickly, with no formal theory or mathematics involved, through pure practical training. These vocational schools are successful in job placement. But it is unimaginable that a 6-month crash course or even a 2-year vocational school could prepare an engineer who goes on successfully to work on designing, say, quantum computers without ever having studied quantum physics or advanced calculus.

No mathematical formalism to guide the development of software

Most books on software engineering contain no formulas or equations, no mathematical derivations of any results, and no precise definitions of the various technical terms they are using (such as "object-oriented" or "software architecture"). Some books on software engineering even have no program code in them - just words. These books talk about how programmers should approach their job, how to organize the work flow or the code architecture, in vague and general terms: "code is about detail", "you must never abandon the big picture", "you should avoid tight coupling in your modules", "a class must serve a single responsibility", etc. Practitioners such as R. C. Martin never studied any formalisms and do not think in terms of formalisms; instead they think in vaguely formulated, heuristic "principles".

In contrast, there is not a single textbook on mechanical engineering or electrical engineering that doesn't have a significant amount of advanced mathematics in it.

Donald Knuth's classic textbook is called "The art of programming". It is full of tips and tricks about how to program; but it does not provide any theory that could guide programmers while actually writing programs. There is nothing in that book that would be similar to the way mathematical formalism guides electrical engineering or mechanical engineering designs. If Knuth's book were based on such formalism, it would have looked very differently: we would first study some theory and then apply it to start writing code.

Knuth's book provides many algorithms. But algorithms are similar to patented inventions: They can be used immediately without further study. Algorithms are not similar to theory. Knowing one algorithm does not make it easier to develop another, unrelated algorithm. In comparison, knowing how to solve differential equations will be applicable to thousands of different problems in science and engineering.

A book exists with the title "Science of programming", but the title is misleading. It does not propose a science, similar to physics, that underlies the process of designing programs, similarly to how calculations in quantum physics derive the properties of a quantum device. The book claims to give precise methods that guide programmers in writing code, but the scope of proposed methods is narrow (the design of simple algorithms for iterative manipulation of data), and is very far from a formal "derivation" of programs from specification. (A book with that title also exists, and similarly disappoints.) Programmers today are mostly oblivious to these books.

To be sure, standard computer science courses do teach analysis of programs using formal mathematical methods. Two such methods are the complexity analysis (the so-called "big O notation"), and formal verification. But programs are analyzed only after they are complete. Theory does not guide the actual process of writing code, which goes entirely by trial-and-error, intuition, and guesswork.

The theory that exists in computer science is somewhat analogous to writing a mathematical equation describing the shape of a shoe made by a fashion designer. True, the equation is mathematically rigorous and can be "analyzed" or "verified"; but the equation in no way guides the fashion designer in actually making shoes. It is understandable that fashion designers do not study the differential geometry of surfaces.

Programmers dislike academic terminology

Programmers appear to complain about the words such as "functor", "monad", or "lambda-functions".

Those fancy words used by functional programmers purists really annoy me. Monads, Functors... Nonsense!!!

In my experience, it's only a tiny minority of programmers who complain about this. The vast majority has never heard these words and are unaware of functional programming.

But chemical engineers do not wince at "phase diagram" or "Gibbs free energy" and apparently accept the need for studying differential equations. Electrical engineers do not complain that the word "Fourier" is foreign and difficult to spell, or that "delta-function" is such a weird thing to say. Mechanical engineers take it for granted that they need to learn about "tensors" and "Lagrangians" and "non-holonomic constraints". Actually, it would appear that the arcane terminology is the least of their difficulties! Their textbooks are full of complicated equations and long, difficult derivations.

Software engineers would similarly not complain about the word "functor", or about having to study the derivation of the algebraic laws for monads, - if they were actually engineers. True software engineers' textbooks would be full of equations and derivations, which the engineers would need to learn and use to perform some mathematical derivations required before starting to write code.

It is now clear that we do not presently have true software engineering. The people we employ under that job title are actually artisans. They employ artisanal methods for their work, and their culture is that of a crafts guild.

Towards software engineering

True software engineering would be achieved if we had theory that guides and informs the writing of code, - not theory that describes or "analyzes" programs after they are written.

Mathematical subject matter (aerospace control, physics or astronomy experiments, statistics, data science) does not automatically make the process of programming into engineering. Most data scientists as well as aerospace engineers and natural scientists are artisans at programming.

Software engineers' textbooks should be full of equations. What theory should those equations represent?

I believe that this theory already exists and may be called functional type theory. It is the algebraic foundation of modern functional programming, as implemented in languages such as OCaml, Haskell, and Scala. This theory is a blend of type theory, category theory, and logical proof theory. It has been in development since late 1990s, and is still being actively worked on by a relatively small community of academic computer scientists as well as software practitioners.

To appreciate that functional programming, unlike any other programming paradigm, has a theory that guides coding, we can look at some recent software engineering conferences such as Scala By the Bay or BayHac, or at the numerous FP-related online tutorials and blogs. We cannot fail to notice that much time is devoted not to writing code but to a peculiar kind of mathematical reasoning. Rather than focusing on this or that API or algorithm, as it is often the case with other software engineering blogs or presentations, an FP speaker describes a mathematical structure - such as an "applicative functor" or a "free monad" - and illustrates its use for practical coding.

These people are not some graduate students showing off their theoretical research; they are practitioners, software engineers who use FP on their jobs. It is just the nature of FP that certain mathematical tools - coming from formal logic and category theory - are now directly applicable to practical programming tasks.

These mathematical tools are not mere tricks for a specific programming language: they apply equally to all FP languages. Before starting to write code, the programmer can jot down something that resembles calculations in abstract algebra, which will help design the code fragment they are about to write. This activity is quite similar to that of a modern engineer who first performs some mathematical calculations and only then embarks on a real-life design project.

A concrete example of this hand-in-hand development of the "functional type theory" and its applications is the so called free applicative functor construction. It was first described in a 2014 paper; a couple of years later, a combined free applicative + free monad data type was designed and its implementation proposed in Scala as well as in Haskell. This technique allows programmers to work with imperative computations where some parts can be done in parallel, and to achieve the parallelism automatically while keeping the composability of the resulting programs. The new technique has distinct advantages over using monad transformers, which was the previous method of solving this type of problem.

The free applicative + free monad combination was designed and implemented by true software engineers. They first wrote down the types and derived the necessary algebraic properties; the obtained results directly guided the programmers about how to proceed writing code.

Another example of a development in "functional type theory" is the so called final tagless encoding of data types, first described in 2009. This technique, developed from category theory and type theory motivations, has several advantages over the free monad technique and can improve upon it in a number of cases - just as the free monad itself was designed to cure the issues with monad transformers. The new technique is also not a trick in a specific programming language; rather, it is a theoretical development that is available to functional programmers in any language (even in Java).

This example shows that we may need several years of work before the practical aspects of using "functional type theory" are sufficiently well understood by the functional programming community. The theory is in active development, and its design patterns - as well as the exact scope of its theoretical material - are still being figured out. If the 40-year gap hypothesis holds, we should expect "functional type theory" to become mainstream by 2030.

Do we need software engineering, or are the artisans good enough?

The demand for programmers is growing. Software developer is #1 best job in the US now. But is there a demand for engineers, or just for artisans?

We don't seem to be able to train enough software artisans. Therefore, it is probably impossible to train enough engineers, especially given that modern Computer Science courses do not actually train software engineers in the true sense of the word (they train academics who are software artisans). The few software engineers we do have are self-taught. Recalling the situation in construction business, with a few architects and hundreds of construction workers, we might also conclude that only a few software engineers are required per hundred software artisans.

What is the price of not having engineers, of replacing them with artisans?

Software practitioners have long bemoaned the mysterious difficulty of software development. Code "becomes rotten with time", programs grow in size "out of control", and Microsoft's operating system has been notorious for its security flaws despite many thousands of programmers and testers Microsoft employed. I think this shows we are overestimating humans' artisanal creative capacity.

It is precisely in designing very large and robust software systems that we would clearly benefit from true engineering. Humanity has been building bridges and using chemical reactions, solving engineering problems by trial, error, and adherence to tradition, long before mechanical or chemical engineering disciplines were developed, based on scientific theory. But now, using the available theory, we are able to create unimaginably more complicated structures and devices.

For building large and reliable software, such as new mobile or embedded operating systems, or distributed peer-to-peer trust architectures, we will most likely need the sharp qualitative increase in productivity and reliability that can only come from transforming artisanal programming into a proper engineering discipline. Functional programming is a first step in that direction.