Success in innovative/complex software systems

The human mind has evolved over tens of thousands of years to avoid pain/discomfort and to seek pleasure. This was an excellent survival trait when food was scarce and the risk of dying in the coming hours or days was very real, but in our modern and much safer world, it has resulted in complex problems or innovation opportunities often being considered "too hard" and thrown in the corresponding basket of inaction. The issue is that problems don't magically go away when you ignore them. It's only by facing up to them, embracing their discomfort and complexity, and progressively breaking them down and understanding them, that they are eventually resolved. The resultant "clear air" is inhaled, the people involved accelerate forward, and the broader community benefits.

I have been fortunate to witness first-hand and be involved in complex problem-solving and innovation "miracles" several times in my 25-year career in software engineering. The successful approach to innovation or "complex" (i.e. new, bespoke, complicated, and/or multi-system) system development is counter to the traditional project management practices that have been inherited from more traditional industries - those where rinse and repeat with detailed plans are par for the course. Ideally, we want to know exactly how long it's going to take and what exactly will need to happen, when, and by whom. With complex software engineering problems riddled with unknowns, it's simply not possible to know these details up-front. You progressively uncover them as you go, and the journey is not linear. Add to this the accelerating rate of change, particularly in software and technology, and the problem domains and solutions are becoming even more complex.

The nonlinearity and lack of time/cost certainty lead to the second issue (the first being the aversion to dealing with complexity in the first place), in that management are understandably risk-averse and loathe to commit valuable resources to uncertain endeavours. It's frequently only when the cost of not resolving complex issues becomes existential that they are given the go-ahead. Even then the response can be delayed and the problem further exacerbated (the software equivalent of climate change). 

One of the most tremendous improvements in dealing with complex systems has been in "Agile" approaches, where we incrementally explore, review progress and understanding, and pivot as we go. For those that have experienced Agile in its truest form, its superiority for efficiently developing complex systems is clear. Unfortunately, this approach is often misunderstood and bastardised into "waterfall with iterations" in an attempt to provide detailed expectations of specific results at specific times that will neatly fit into quarterly reports. What typically ends up happening in these situations is the team avoids the crux of the problem (due to its higher risk and uncertainty) and instead tinkers around the edges to provide a workaround that masks the underlying issue for some of the time (in the hope that this will be enough). This is highly inefficient as cracks inevitably appear and the team progressively addresses the other situations where the issue is still a problem until it reaches a point where they need to solve the underlying problem and effectively start again from scratch; meanwhile having wasted vast time and resources in the process. The other approach is where the team believes (often with good reason) that the importance of the problem is so great and not duly understood by the organisation that they tackle the problem by stealth, unbeknownst to the broader organisation. This leads to conflicts in priorities, frustration in delivery timing and coordination, and potential burnout in members of the team.

Neither of the two aforementioned approaches is ideal. I believe the most-effective and efficient approach for tackling complex systems involves building the necessary trust and communication/collaboration between the various parties involved and using an incremental approach (which are incidentally key values of agile). This relies on a suitably strong culture of trust and collaboration within the organisation that rewards such behaviour and holds people accountable (at all levels). Everyone is responsible for living the culture, but it must be top-down driven and exemplified to be successful. Conversely, when the motivations and behaviours of the parties involved don't align (e.g. wanting to impress higher-ups with promises of a "quick fix") things can quickly fall apart.

So where does all this leave us? In summary, to be successful in complex/software engineering systems IMHO requires: 

1. self-awareness and courage to face complex problems head on

2. Agile approaches to incrementally build understanding, share learnings, and review priorities

3. a strong culture of trust, communication and collaboration with aligned objectives

I hope you've found this helpful. I would love to know what has worked for you.