Beyond Metrics!
The article is writer's own perspective and is NOT related in any ways to organization's perspective, writer belongs to.
Introduction
DevOps is one of the key trends in the industry today. While there are quite a few organizations that have made significant stride in the way they deliver software, majority of the organizations are still at the beginning of their journey. One of the important elements to be successful in this journey which is often ignored is the ability to effectively measure the progress.
In the last few years we have seen multiple reports on DevOps assessment with the most popular being DORA that talks about state of DevOps across the enterprises. DORA metrics (Lead Time, Deployment Frequency, Mean Time to Resolution & Failure Rate) have now been adopted by multiple multiple DevOps tool chain as a way to measure the throughput and stability. You will see some variation of these metrics in tools/services like Harness.io, GoCD, DevOptics etc.
It definitely is a great starting point in my view, but there is plenty more to explore. So what could we do? In this article I take the fundamental of DevOps as a pivot, that can also be used to measure its progress as well. At the end of the article I share few potential scenarios. Let's first define DevOps
"DevOps is the union of people, process, and products to enable continuous delivery of value to our end users."
A pictorial representation of a typical DevOps workflow
People, Process, Products
If DevOps is a union of people, process and products then shouldn't we measure the progress of the journey on all three fronts?
Current available metrics are limited to product and do not exploit the People and Process aspect. They are good at surfacing information about the tool effectiveness based on the data captured during usage. Before we delve into each of these in more details (People, Process and Product), let's look at what these three terms could relate to
People: This is really what forms the culture of the team
Process: The first thing that comes to the mind is laborious, documentation, paper work, however when we think about process we should think about things that can act as enablers for team.
Products: Includes the tools and services used to achieve the desired goal. It is a medium to achieve the desired goals.
Scenarios
Let's look at some scenarios under each category
Product
Measuring product efficiency and reliability is focused towards the tool and services being used for DevOps. E.g. do you have enough machines to meet the capacity needs for builds, tests, deployments? Are these environments reliable? How often do we observe failures during deployment? etc. Let's look at a few hypothesis around Product
Hypothesis: "Providing insights about failure patterns in the pipeline based on their similarity, will help teams prioritize and troubleshoot failure faster thereby improving productivity"
Here is an example of a deployment reliability report. An engineering director can look at the health of the micro-services for reliability of the deployments and drill deep into the services that needs attention
While the engineering director is looking at the overall health, a developer is interested to understand the categories/patterns of failures in the DevOps pipeline using failures bucketed based on similarity of logs. These is a great example of Machine Learning complimenting DevOps.
Let's take a look at one more hypothesis focused on efficiency
Scenario 2: "Insights into pipeline execution duration will help team identity the change-points and take preventive measures"
In this specific case, the developer can see all the points where the duration of execution of a DevOps pipeline and its tasks changed.
Scenario 3: "Identifying non-deterministic test failures called flaky tests can help improve developer productivity by reducing the debugging and troubleshooting time."
Process
Process scenarios as mentioned above are focused on identifying missing best practices that if identified can act as productivity enablers for the team. Let's look at some hypothesis on Process
Hypothesis: "Detection of test stages in the DevOps life-cycle and it's association with the failure rate can provide insights into need for left shift quality"
Hypothesis: "Detection of existence/non-existence of the PR process coupled with the ability to measure PR quality can improve the quality and PR turnaround time"
People
People scenarios focus mainly on the cultural aspect within the team. This is also one of the hardest to measure, however not impossible. We will take a look at few hypothesis around this area
Hypothesis: PR workflow can provide deep insights into team collaboration and sentiment
Hypothesis: Customer issue sentiment can act as a representation for team health as well as customer centered behaviors.
Conclusion
There are a lot of other scenarios that fall into each of these categories, some overlapping others distinct, however in my opinion the complexity of these scenarios increase from Product to Process to People. To conclude I would like to re-iterate that is People, Process and Product are the pillar of DevOps then to effectively measure the progress of this journey one needs to look at it from these three lenses.
Thoughts ? Inputs? Feedback?
