The System Was Never the Code

There is a particular kind of confidence that surrounds modern software.

It is the confidence of generation. Of production at speed. Of systems assembled through prompts, stitched together by models, deployed before they are fully understood. The interface is clean, the feedback loops are immediate, and the outputs—at least initially—appear coherent. For a moment, it feels as though the problem of software has been solved. Not by better engineers, but by better tools.

And yet, something quieter is happening beneath this surface acceleration.

Systems are becoming harder to understand at precisely the moment they are becoming easier to produce.

This is not a contradiction. It is a structural shift.

The Disappearance of the Program

In 1985, Peter Naur wrote a paper with an unassuming title, Programming as Theory Building. Its central claim was simple, but it has aged into something far more disruptive than it first appeared:

The program is not the code.

At the time, this was a corrective. Programming was increasingly being framed as a formal, almost mechanical activity—something that could be specified, verified, and executed with mathematical precision. Naur pushed back. He argued that what actually sustains a system is not its formal correctness, but the coherence of the understanding behind it.

Code, in this framing, is a trace. A residue. A shadow cast by a deeper structure of thought.

Remove that structure, and the code remains—but the program, in any meaningful sense, does not.

For decades, the industry absorbed this insight without fully confronting it. The friction of programming—its slowness, its difficulty—acted as a kind of informal safeguard. To build a system, you had to spend time inside it. You had to wrestle with its edge cases, its dependencies, its failure modes. The theory was not optional. It was the cost of entry.

That cost has now collapsed.

When Production Becomes Cheap

Generative AI does not eliminate programming. It eliminates the need to construct programs in the traditional sense.

Code can now be produced on demand. Entire features can be scaffolded in seconds. Patterns that once required years of experience can be invoked through a sentence. The bottleneck has shifted decisively from production to evaluation.

This is often framed as a familiar story. Another abstraction layer. Another step in a long progression from assembly to high-level languages to frameworks. But this analogy does not hold.

A compiler is a deterministic system. It translates input to output in a way that is consistent, verifiable, and ultimately trustworthy. Its internal complexity can be safely ignored because its behaviour is constrained.

A large language model is not that kind of system.

It is probabilistic. Generative. Context-sensitive. It produces outputs that are plausible rather than guaranteed. The same input can yield different results. Errors are not exceptions; they are part of the operating model.

This is not an abstraction you can rely on without understanding. It is a collaborator you can only use effectively if you understand.

The distinction matters. Because it changes where responsibility sits.

The Return of the Theory

If Naur was right—and there is little reason to think he wasn’t—then the real question is not whether AI can write code. It is whether the theory of the system is still being built, and where it now resides.

In practice, it often isn’t.

What emerges instead is a pattern that looks increasingly familiar across domains. Systems that function in the narrow sense—they run, they respond, they scale—but lack a coherent underlying model. State is duplicated or ambiguously owned. Feedback mechanisms are partial or absent. Dependencies are implicit rather than understood.

These systems do not fail immediately. They degrade. Quietly at first. Then suddenly.

What appears as a technical issue—a race condition, a broken integration, a scaling bottleneck—is often something else entirely. A failure of understanding that was present from the beginning but deferred by the ease of production.

This is what is now being described, somewhat loosely, as “comprehension debt.” But the term understates the problem. It suggests a temporary imbalance, something that can be repaid with time and attention.

What is actually accumulating is not debt, but fragility.

Institutionalising the Problem

This dynamic is not unique to software. It is structural.

In Institutional Maturity Theory, systems are not evaluated by their outputs alone, but by how they embed responsibility, maintain competence, and adapt to complexity over time. The artefacts of a system—policies, models, codebases—are secondary to the question of whether the system’s understanding is distributed, maintained, and recoverable.

From this perspective, Naur’s insight scales cleanly.

The “theory in the programmer’s head” becomes a question of information embedding. Who holds the model? Where does it live? Can it be reconstructed when individuals leave?

The ability to work with that model becomes competence. Not the ability to produce outputs, but to interpret, challenge, and adapt them.

And the alignment between what is built and what is intended becomes responsibility. The system must not only function; it must be owned.

When these dimensions weaken, systems enter a familiar state. They continue to operate, often for long periods, but their internal coherence deteriorates. Decisions become harder to justify. Changes become riskier. Failures become more opaque.

The system, in effect, loses its theory.

AI as an Accelerant of Immaturity

Generative AI does not create this condition. It accelerates it.

By removing the friction of production, it removes one of the primary mechanisms through which understanding was historically formed. The struggle to implement a feature, to debug a failure, to trace a dependency—these were not inefficiencies. They were part of the learning process through which the system’s theory was built and internalised.

Without that process, it becomes possible to construct systems that are functionally complete but conceptually hollow.

At the same time, AI introduces a second-order effect. It diffuses responsibility. When code is generated by a model, the line between author and tool becomes blurred. Errors can be attributed outward. Decisions become less explicit. The system, as a result, has no clear locus of accountability.

This is a classic maturity failure.

High output, low embedding. Rapid adaptation in form, limited adaptation in understanding.

The system appears advanced. It is, in practice, brittle.

The Collapse of the Pipeline

There is a further consequence, now becoming visible at an industry level.

The traditional pathway from junior to senior engineer was not simply a matter of time served. It was a process of exposure to failure. Systems designed poorly, systems that broke under load, systems that had to be rebuilt under pressure. These experiences formed the basis of the “theory” Naur described.

If that pathway is disrupted—if juniors are shielded from the need to engage deeply with systems because tools can fill the gap—then the formation of that theory is delayed or prevented.

What emerges is a gap. Not immediately visible, but structurally significant.

Organisations find themselves able to produce more software with fewer people, but less able to maintain, extend, or reason about what they have built. The pipeline that once produced system-level thinkers begins to erode.

The response, predictably, is reactive. Hiring patterns shift. The need for oversight reasserts itself. The system attempts to rebuild the capability it has inadvertently degraded.

But the underlying dynamic remains.

The New Constraint

If production is no longer the constraint, then something else is.

The constraint is now coherence.

The ability to hold a system in the mind. To understand where state lives, how feedback propagates, what fails when a component is removed. To make decisions not just about what can be built, but what should be built, and how it will behave over time.

This is not a new skill. It is an old one, reframed.

What has changed is its position in the hierarchy. It is no longer the endpoint of experience. It is the entry requirement.

Rebuilding the Theory

The question, then, is not whether systems thinking matters. It is how it is cultivated in an environment that no longer enforces it.

There is no clean answer. But the direction is clear.

Understanding must become deliberate.

Design must precede generation. Not as a formal requirement, but as a cognitive discipline. The system must be sketched, articulated, challenged before it is produced.

Specifications must re-emerge, not as bureaucratic artefacts, but as scaffolding for thought. The articulation of intent, constraints, and failure modes becomes the anchor against which generated outputs are evaluated.

And perhaps most importantly, systems must be interrogated. Components removed in thought. Dependencies traced. Assumptions surfaced. The theory must be actively maintained, not passively assumed.

These are not new practices. They are reintroductions of constraints that the tools have removed.

The System Was Never the Code

There is a temptation to frame this moment as a technological turning point. A shift in tools, in workflows, in the economics of software.

But the deeper shift is epistemic. It concerns what it means to know a system.

For a long time, the act of building and the act of understanding were tightly coupled. To produce was to learn. To implement was to internalise.

That coupling has now been broken. What remains is a choice.

Systems can be built without theory. They can function, scale, and even succeed in the short term. But they will carry within them an absence. A gap between what they do and what is understood about them.

Or the theory can be rebuilt, deliberately, under new conditions. Not as a byproduct of effort, but as a primary objective.

Because Naur’s insight still holds. The system is not the code. And it never was.