The High Cost of Ignoring Software Deployment Risks

This week, the global realization of several risks in software development and deployment came into sharp focus. A recent update from CrowdStrike brought the #BSOD front and centre, sparking global media claims of an 'internet outage'.

This resonates with points I recently discussed in response to "IT Underperforms Again, Again" by Prof. Bent Flyvbjerg , where I noted: "...risk management is often ineffective, with senior stakeholders willing to embrace substantial risks. The real challenge arises when they are reluctant to accept the consequences if these risks materialise..."

In the development and release chain, numerous risks are often accepted, assuming a low probability of occurrence. We understand that monitoring and security products operate at critical system levels to prioritize access and protect the system. However, errors at this level can cause severe, unrecoverable failures—yet antivirus and security updates are often considered low-risk deployments when perhaps they are high risk, low probability.

For such a widespread problem to have occurred there must have been systematic failures in multiple places.

1. Crowdstrike, The release might not have undergone adequate testing: While it's difficult to know the exact level of testing without insider information, one would expect such a prevalent issue to have been identified before release.
2. Customers, Accepting a release from a vendor and pushing this directly to their estate: Although there is a general expectation that vendors release high-quality, thoroughly tested software, companies vary greatly in their release management strategies. Risk-averse organisations likely test and manage releases extensively, while others, more accepting of risk, might directly push updates, focusing only on testing new features.

So how do we mitigate these risks? Vendors will often use an early release 'eat our own dog food' approach, releasing all software to their teams ahead of public release. It's better to cause an outage of your internal systems than wipe $16B off your market cap.

This approach really should be deployed throughout the entire development journey from beta to general availability. Microsoft 's ring deployment methodology suggests a phased rollout to mitigate such risks, a strategy used across Windows and Office ecosystems, allowing users to choose their release track. If a similar methodology had been implemented, the scale of the problem could have been significantly reduced, likely affecting only non-critical systems.

In cases where complex race conditions make problems hard to pinpoint, some issues can be excusable. However, when a significant number of Windows machines have critical failures due to a software update, it's a clear indication of poor risk management.

Photo by Loic Leray