Abstracting Engineering Away

Should programmers call themselves engineers? Controversy aside, I think there is a real, meaningful distinction to be found: programmers and engineers deal with different levels of uncertainty.

By uncertainty I mean: if you want to solve a problem, how well do you know what you are doing and how will you do it?

For example, researchers know very little about what they are doing, and even less about how they'll do it. They first need to figure out the right questions to ask, and only then the answers. In that sense, researches deal with high levels of uncertainty.

Unlike researchers, engineers have a better understanding of what they need to accomplish. For example, build a nuclear reactor. Design decisions are needed: should it be a boiling water reactor or a pressurized water reactor? Engineers face well-understood trade-offs and can reference previous solutions. Clearly, that doesn't make building a reactor an easy task. Expertise, skill, and effort are needed to implement any complex solution, even if is well-understood.

Finally, technicians implement known solutions for known problems with little variation. An electrician can estimate the fuses required for a house, install them, and rarely get it wrong. When the problem is well-understood and the solution is repeatable, there is low uncertainty.

The three categories, researches, engineers, and technicians, offer a continuum of technical activities with varying degrees of uncertainty. Where does programming fall in the scale?

Most people assume it neatly falls alongside engineering: its members call themselves "software engineers". Thus, people expect software to work like any other engineering. This leads to all sorts of confusion, most notably with regards to scheduling and failure rates.¹ Society's expectations correlate with uncertainty: research is not held to deadlines and failure is accepted, engineering projects are scheduled with some leeway but expected to succeed, and technical tasks are integrated as predictable steps into larger processes.

While there is a subset of engineering that resembles programming and a subset of programming that resembles engineering², uncertainty is higher in programming.

Let's pick up our very serious example: building a nuclear plant. The engineers will use the best practices accumulated from previous engineers. Their job is to check how does this current plant differ from previous ones and adapt the known methods. But what happens if there are no previous plants?

The Borax experiments were carried by the team that built the first nuclear reactor in the US. They faced unexplored requirements and constraints. They had to do much more testing and prototyping than current nuclear engineers. They also probably felt like they didn't know what they were doing, because in fact, they didn't. Did they "act like engineers" or like "sloppy programmers"?

These Borax-I experiments were expected to provide proof as to whether or not a water reactor could operate in a stable boiling mode. ... It began as a crude reactor experiment, conducted in open air without benefit of any building structure or containment.

Even nuclear engineers skip SSL on the prototype. Clearly, engineers have to experiment at some point. With that in mind, the following quote makes sense:

Reactor operators, at that time, had only on-the-job training. It was a bit like how Orville Wright learned how to fly. ... The loose arrangement allowed for decisions and problems to be resolved quickly, sometimes with whatever materials were conveniently available.

Translation: they winged it and used duck tape.

There was one commentator who described the Borax experiments as having been conducted by a bunch of cowboys. It was true that we knowingly accepted greater risks than would be acceptable today. ... There was insufficient time available to make the needed modifications. We compensated for the problem by implementing some very innovative tricks.

Yesterday's tricks are today's best practices and engineering is a collection of tricks that are known to work.

So why is it that all programming feels like the experience of the first nuclear engineers? Abstraction and reuse. Software is free to copy and can be abstracted for reuse. When the software industry solves a problem, it solves it for every project. For example, when was the last time you implemented your own HTTP server? On the other hand, civil engineers need to pour concrete for every building, even though the Romans had a great recipe for it.

When starting a new project, programmers only face the novel elements. For any know problems, they can reuse somebody else's code. Here are some solved problems:

• Communicate arbitrary data over a network (IP/TCP).
• Communicate human readable documents over a network (HTTP).
• Store data with "table-shape" (SQL databases).

Here are some of the problems not yet solved:

• Build distributed systems that behave as expected.
• Deploy and manage clusters of servers and applications.
• Process large amounts of data either in batches or continuously.

Distributed systems don't look like something you can abstract over but more like a design space. Accordingly, people dealing with them increasingly sound like engineers. They refer to trade-offs (consistency vs availability) and recommend best practices ("don't write your own consensus protocol!"). Similarly, specialized software consultants behave like engineers. They specialize in niches like integrations or payments and see the same problems over and over. They run the risk that their entire niche could be abstracted away at any moment.

There are other programming activities with lower levels of uncertainty.³ Consider the average "This business exists!" website. Web consultants can reliably use HTML, PHP scripts, and Wordpress. It's been done a million times, there are few surprises and little need for investigation. They are more like technicians than engineers, implementing a repeated solution for a very well known problem.

When programmers solve a problem, abstraction lets them encode best practices and the industry as a whole moves on to the next frontier. Once a problem is abstracted away, implementing all future solutions will feel like plumbing and not like engineering. The next problem is by definition new so it doesn't feel like engineering either!

Abstraction and reuse are not restricted to software. Electrical engineering has such pervasive standards that it almost feels like no design is needed. There is only one thing to produce, AC electricity, so components can be easily "connected". Electronics has more powerful abstractions: individual components (e.g., transistors) can be composed into more complex components almost indefinitely (computers!). That is one of the reasons digital electronics are more powerful than analog electronics: analog components can't be composed indefinitely because of noise propagation. Mechanical engineers have a harder time. Pumps, electrical engines, and small combustion engines have been abstracted into catalog components. Beyond that, things are harder to abstract: you need to redesign support structures and stress test for each project. Civil engineers face the hardest challenges when dealing with reuse but even they've found some abstractions: steel framing for skyscrapers and prebuilt homes transformed "building a structure" into "assembling a structure" which is much quicker.

All those disciplines have distinct places where the quirky engineers tinker and explore: workshops. If you believe that engineers are somehow serious, visit a workshop. You'll either leave thinking that tape is a valid structural material or you'll never trust an engineer again. While the tinkerers remain hidden in workshops, the rest of the engineers are visibly busy solving problems they couldn't abstract away: keeping the production line running, pouring concrete into molds, etc. In programming the opposite is true: the tinkerers are the majority, goofing in public.

Engineering only emerges when a problem has no final answer but actually represents a design space with multiple solutions and trade-offs. Programmers have been around for less than a century and are still abstracting problems away. Let's hope it continues that way, tinkering is much more fun.