The Inevitability of the Solution

There's No Such Thing as Eureka!

As a so-called "software architect", I wear many hats. One of my favorites is 'developer' because the ultimate job satisfaction is in making things that provide value back to the business. Of course this is a form of myopia: there are a ridiculous number of "moving parts" in software engineering disciplines between "I wrote some code" and "my code is in production". Many of them get little acknowledgement outside the core engineering group, but nearly all of them are vital, and the list of what is vital keeps growing.

Technology doesn't exist in a vacuum of course, it exists because a business (or government, or military) needs a better way to do something. As technologists it is an easy vanity to fall back on to believe that there is some higher ideal, some purity we strive to achieve that exists outside of those core business needs. This notion is probably not entirely false, but giving it any play in foresight is where that vanity comes in. This higher ideal, purer solution thingamabob can really only exist as a function of hindsight. 

As a so-called "software architect", I strive for simplicity. It is far too easy today to make a system complicated unnecessarily, adding no real value. It is also easy to ignore non-functional requirements and pretend at simplicity while failing to create a system that can support the appropriate capacity. What I often struggle with is where to draw the line. I regularly consider in the design process how to honestly support the anticipated requirements of the system, while both not introducing artificial constraints as well as not introducing unnecessary complexity in support of unnecessary capacity. It seems every newfangled thing out there is designed for supporting billions of hits, yet how often is this actually the business concern? Why pretend it is? Just don't pretend it isn't...understand the business need.

In short, I try not to use dynamite to manage a gopher problem. It is terribly easy today to put too much thought into the solution, and terribly easy to just lean on cloud-vendor solutions without a consideration of the appropriateness.

An example: Do I need Apache Spark to open 200 files with 10K of data once a month and process some data into a RDBMS? Of course I CAN use it. But should I? How much overhead, both in dollars and people am I placing on the business by introducing a complex tool?

More often than not, I find the right solutions sort of pop out naturally after working the problem as a thought exercise a couple of times. When I spend a bit of time actually thinking about the problem, it usually allows me to draw on prior experience as to when this or that solution actually served me in the past, vs. creating more problems. When I start with writing code for proof-of-concept work, I often get married to a solution regardless of its correctness because hey, I got something working. That bit of reward is like the cheese at the end of the maze. The mouse only wants to run the same maze again. Known effort. Known reward.

As I work through these thought exercises, patterns from my past often begin to emerge. I'm talking about conceptual aggregates here, not tools. Imagine shadowy venn diagrams of how this and that process ended up being related, or what components of them were useful.

Innovation is the result of intensive work. Eureka moments are usually identified in the process of reflection, not prediction. Software architecture solutions are the manifestation of experience in trial and error, frustration, and more trial and error, all bottled up and saved for the next design need. 

Back in 2010/11 I worked on a project where we had become tired of the complexity in the classic "lasagna architecture" where everything has its place in a layer and you don't jump layers (controller talks to session facade talks to service talks to dao - this sort of thing). Realizing that we had these giant interconnected pieces in the code, we started to disconnected and simplify, and create small deployable units that we could aggregate under a single build and run in a single app server. This was almost-microservices before any formal definition was in place. Months later I read about microservices, then Fowlers paper came out, and then I was "doing microservices" regularly in organizations.

Regardless of if you are a fan of this design approach, or in the camp that is slowly moving away from it, one cannot argue that it was seen as a panacea for the monolithic complexity that had become the norm in software.

There are many other examples of this, NoSQL (alternative storage and access), Hadoop (large scale batch processing), continuous delivery - in any of these I can point to a specific turning point in my own thinking and solutioning, in advance of a particular mainstream solution coming into being. I do not credit myself with any particular genius in this recurring theme of my career - of course I would love to! But the truth is more likely that these solutions are the next naturally occurring designs that arise from the current state of tooling and evolution of business needs. I think of non-software examples in history where the co-occurrence of invention took place: Calculus, Television, the Microchip, the jet engine...etc, and realize this is not altogether strange, nor even particularly brilliant.

The state of our tools, the corpus of our knowledge, the mechanisms of our approach combined with the goals of our mission, all contribute to a sort of trial and error where the ultimately adopted solution becomes inevitable.

I suspect this is just a bit more frequent in the software industry due to the pace of development and adoption.

As a so-called "software architect", I continue to borrow where I can, and invent where I have to. Yesterday it was Devops, today it is Dataops. Automation isn't new. Cross-team communication and collaboration aren't new. We are simply finding the next steps in an N step process that make the most sense given the conditions we are living under, and collaboratively attempting to improve.