DevOps - How to do Governance at Hyperspeed

This series has so far covered What DevOps is and Why you should care.

Now we’ve got a handle on the basics, it’s time to tackle one of the most common questions about DevOps – how do you maintain ‘control’ in such a dynamic environment?

Or, in other words, how does governance work? Hang-in there, this isn’t as dull as it sounds!

What is Governance anyway?

Governance is the method by which a company ensures that the work being done forwards its strategic goals and targets.

Clearly this is essential whether you are doing DevOps or not. After all, there’s no level of agility that avoids the need for clarity of purpose and direction.

Whilst there are lots of interesting aspects of governance including budgets and compliance, this article is going to focus on change management to highlight the contrast between the two approaches.

Let’s get into it.

Traditional Governance

I once submitted a request for a new laptop a full month ahead of a contractor starting for a short engagement. 59 days later, on the contractor’s very last day, the laptop showed up. Misconfigured.

I saw another company spend £100,000 on new hardware simply because getting changes approved for the existing servers was virtually impossible and they didn't have time to follow the process.

Sadly, these are not isolated examples. In many big businesses, the term governance is associated with a staggering confusion of documentation, a magical mystery tour of stakeholders, an impenetrable torrent of flow charts and the kind of wanton meeting abuse that keeps an industry of therapists and pubs in business.

The cumulative effect is that getting things done becomes an exercise in persistence and frustration. Not only does this slow everything down, it snuffs out the sparks of innovation before they can get started – simply because it’s not worth the hassle to try.

You’ve probably experienced this kind of governance, but how do things get like this?

The issue is in how the systems grow over time. Because there is a governance machine in place, it becomes the easy (lazy) home for every new control requirement. Once added, controls are almost never removed. This is partly because their original purpose gets lost in the mists of time, but also because questioning those with the power to cause you problems is a naturally self-limiting activity.

As governance structures get more complicated, their sheer weight discourages frequent use.

This leads to changes being batched up so teams don’t have to spend so much time jumping through the hoops. Of course, bigger changes are riskier meaning more things can go wrong – leading to yet more process when they do!

You have to admire the self-fulfilling nature of this kind of governance. It needs to be heavyweight to support the fact that being heavyweight causes so many problems.

DevOps Governance

OK, that was a touch depressing. What’s the alternative then?

The biggest difference with the DevOps approach is this: you make small changes. One at a time, pushing them all the way through to live using continuous delivery.

In this context, a ‘small change’ is a single story, bug fix or task assigned to the team.

Many DevOps practices follow logically from this principle. Just think about what you would need to sustain a stream of continuous releases:

• Your deployment and rollback mechanisms would need to be automated. This removes manual and error prone steps making them faster, more consistent and more reliable.
• Your testing needs to be automated so that tests can be run quickly and frequently as part of every release.
• You need to be able to monitor in real-time whether a release has caused any unintended impact so you can trigger a rollback
• Rollbacks need to quick and clean to avoid delaying or disrupting the next release. This implies the use of blue/green deployments or similar patterns running on flexible Cloud infrastructure.

The somewhat counter-intuitive result is that overall risk goes down - despite changes being made more often. This is because each change carries only a nominal risk thanks to being small, specific, well understood, isolated, easy to test, easy to deploy, well monitored and easy to rollback.

If this isn’t true, then you fix those things - rather than adding more controls to a centralised system to compensate.

So, what does this mean for change processes?

Well first, it means they need to be simple so they can be run frequently with low friction. Fortunately, with less risk to mitigate, it’s easier to keep controls focused and maintainable.

Simple controls are also easier to implement – often able to be enforced directly through automation. This is a great way to distribute knowledge and inject governance feedback where and when it is needed most.

In fact, automation is the key to governing effectively at scale and speed.

Aside from the obvious benefits, automation also provides a platform for capturing better metrics and improving the visibility of process progress and effectiveness. For example, seeing what changes have been made across environments and what the results were. This insight is invaluable for driving continuous improvement and giving your teams the situational awareness necessary for good decision making.

Automation also helps capture unambiguous deployment documentation (i.e. the deployment code) and provides an audit of deployment changes via source control. This is a boon for security and compliance. If all changes are pushed through automation and server logs get consolidated, why would anybody need access to live servers anymore?

With simple governance rules supported by automation, responsibility for release approval can be safely delegated to engineering teams. This removes central bottlenecks and empowers your teams to work at their own pace, releasing in a controlled, audited and visible way as and when they are ready. This creates a much more scalable model.

More importantly, it sends a signal that governance is everybody’s responsibility. Rather than doing the minimum necessary to pass the requisite tick-boxes and throwing the result over to a different team, everybody is encouraged to care about quality. Because cross-functional teams run the systems they build, they feel the effects of any quality issues and governance decisions first-hand.

Summary

Traditional change governance is almost always a central point of contention and friction. A lot of it is only necessary to mitigate the risks inherent in making big, occasional, waterfall-style changes.

DevOps governance is based on the agile principle of making small changes regularly. It aims to proactively tackle the underlying issues that make production releases risky in the first place - rather than compensating for these with extra process.

Supporting this requires an investment in automation as well as the distribution of governance responsibilities across empowered teams. Done right, this allows governance to be lighter, faster and more scalable without compromising effectiveness.

Better yet, it frees organisations from living in a confusing bureaucracy that slows everything down and stifles innovation - whilst staying true to the original purpose of governance. I think that’s something we can all get behind!

Governance is a big topic and no single blog post can do it justice. So, let me know your thoughts and questions below – what do you think about DevOps and governance?

In my next post, I’m going to talk about DevOps and project management. Specifically, how to square agility with the reality of business deadlines.