The DevOps Hyperbole Machine

In 2016 Puppet Labs produced their latest State of DevOps Report. Since then the report has been oft quoted. In particular almost every marketing driven article about DevOps is reporting that "High performers deploy 200 times more frequently than low performers". 

200 times more frequent deployments sure does seem to be a lot, and being a high-performer definitely sounds like a good thing. But to really understand what these numbers means you need to drill in to the report methodology.

The cluster of companies that deploy the most is labelled high-performers ... we could just as readily call this group frequent deployers.

At the heart of the report is the clustering of survey respondents by the features: deployment frequency, lead time for changes, mean time to recover and change failure rate. 

Three clusters were discovered. The cluster of companies that deploy the most is labelled high-performers, seemingly a reasonable moniker, although an arbitrary one. We could just as readily call this group frequent deployers. But "Frequent deployers deploy 200 times more than infrequent deployers" just does't have the same punch. 

Wait though, these high-performers deploy 200 times more often, but have only a 3 times lower failure rate. So they are failing approximately 66 times more! They recover 24 times faster, but this still seems to suggest that they spend more time in outages, according to the report "...the low-performing group has lower total downtime costs than the high-performing group".

The report goes on to use the high-performers group to elaborate the various positives of DevOps, including higher employee loyalty, less time spent on rework, and so on.

The problem here is that most of these claims are significantly weakened by the lack of demographic information. Demographics are provided for the overall survey but not for the individual clusters. 

DevOps is not a cup to be won by deploying more often than your competitor

Roughly 50% of the companies surveyed have more than 500 IT staff, but how many of these fall in to the high-performers group? Are employees more engaged because they are doing DevOps, or because they working in small small web start-ups? If only 20% of respondents are high-performers, and 90% of these are technology companies then its remiss of us to use this data to draw definitive conclusions about, for example, the finance industry. 

In the end the Puppet Labs report is useful; even if its key findings do sometimes reach beyond the data. The real peril lies in out of context statistics being repeated like a mantra, gaining a life of their own. Lets get one thing straight: DevOps is not a cup to be won by deploying more often than your competitor.