What are Complex Software Systems, and why should I care?

I specialize in engineering complex software systems. Here I'd like to unpack this idea and describe how it may apply to you and your organization.

A hard-to-apply but comprehensive definition of complex SW

One compact definition of a complex SW business goal is -- a goal that transcends what a standard craftsperson can build alone given enough time.

In other words, if you go to a (decent) hiring agency and ask reasonable candidates if they could accomplish that goal while working alone, if more than half tell you they can't do it - boom! you're in Complex SW Land!

Heuristics for SW complexity

Since the above criterium is difficult to evaluate, I wanted to showcase several kinds of complexity that I encountered during my career. Note that even one of those could make your SW complex, and if there are several, they will compound each other.

• Developing code to run within a constrained HW resource envelope, for example: "much more data than RAM". Complexity here is conditional on going beyond the readily available solutions -- addressing the above example using an off-the-shelf database is not complex; developing your own data retrieval layer is. The same example also demonstrates that avoidable complexity is highly undesirable -- if you have a choice between building your own DB and taking one off the shelf, only the former makes sense.
• Any work involving an algorithm whose design requires a proof of correctness/validation.
• You cannot compromise on reliability -- which means that reliability becomes a top-line engineering goal on its own right.
• You need to develop interdependent infrastructural features involving more than 1 person, over a dependency graph that spans more than 3 months.
• You're in maintenance mode (no major architectural dilemmas), but to keep up with the amount of maintenance/operations work, you need 3 or more people.
• You need to interface two software ecosystems -- for example you have a modern cloud-based front-end, but your back-end is an antiquarian on-prem application DB.

Why should I care?

SW Complexity implies a self-reinforcing loop (vicious cycle) which - if unmanaged - may be quite destructive to the organizations developing and using that SW.

Short-term, complexity may lead to a failure of execution - for example, the organization may go over budget on HW, or have to go into a bushfire mode due to frequent service outages which will waste a lot of people's time and cause attrition to the few qualified people around.

Long-term, the price will manifest in those organizations' culture. The need to grow in spite of raging bushfires will mean settling for mercenaries who don't care, onboarding otherwise unnecessary administrative staff, which will introduce its own agendas misaligned with those of the organization at large, which means that the leadership will have to struggle to retain the edge/focus on the organization's main business mission, which is almost never SW per se.

I've seen cases in which improperly managed complexity requires what I estimate as about 3 times as many equivalent engineers as what could be necessary if the culture had been kept in a better shape. Few organizations can afford that.

What can be done?

This is a subject for a whole series of articles (ping me if you'd like to see them written).

Complexity must be accounted for - you need somebody empowered to weigh the technical strategy decisions against the medium and long-term business implications.

Almost invariantly, if only possible, one has to avoid complexity. For example, only in the rarest of cases should an inspired bespoke solution be chosen over an off-the-shelf one.

Finally, only if you must - you need to establish complex SW sub-organization(s), crew them appropriately, set expectations and manage risks.

Sounds challenging? That's good news for me, for it is my daily bread.