Metrics with Design Thinking, Lean, Agile and DevOps

If you can't measure it, you can't improve it. - Peter Drucker. No arguing this. But, what to measure? Does the engineering metrics remain same or it changes as the organization matures on lean principles? How much to measure is also important as too much of measurement can lead to management paralysis. So, selection of metrics has to be well thought through and can’t be taken lightly as it has a deep-seated impact on results and organization’s culture.

Generally, lean journey in an organization starts with agile rollout. Scrum is one of the agile methodologies which is simple and most widely adopted.  In scrum, product development is divided into sprints and a release could constitute of a single or a few sprints. The team measures amount of work they have completed at the end of every sprint as sprint velocity. Velocity is the total of story points for the fully completed user stories in that sprint. Velocity is the throughput of the team but it should not be used as a productivity metric as it is based on story point, which is size of the user story and not the actual effort. Story point size differs from team to team and is not standard across the organization. Sprint Velocity is used by agile teams in release and sprint planning as it provide them an indication of their throughput.

Release burndown or burnup chart are the metrics to track release progress sprint over sprint. It shows how much work is done and how much is remaining to be worked upon in the project. It along with dependency tracker is used to track complex projects and also manage cross-functional dependencies.

Release burndown chart and sprint velocity are two basic metrics that every scrum team maintains and track. A few additional quality related metrics like – bug slippage, test coverage code coverage, and others are used by team and it varies from team to team.

As the organization matures into DevOps and Lean, productivity tracking and productivity improvement become more and more relevant. As I have discussed in my previous articles agile, DevOps and lean are all centered on one common theme which is speed. Speed to market and speed to value matters to the business most, so shouldn’t it be the primary metrics for product engineering? The earn value or effort based productivity metrics and measurement are good for traditional industries, but these are not suitable for knowledge business.  

User story lead time is the time it takes for a user story to get delivered since it was created. Use story cycle time is the time it takes a user story to get delivered since it was picked in the release for development. Lead time includes cycle time.  And to put things into perspective, a lot of the big technology companies can now have a lead time of 1 hour or less.

Speed to value is the time it takes from the customer to realize value since she made the purchase. In a SaaS business, it would ideally be a few minutes whereas in on-premise model it would be a few hours, off-course excluding time required to setup hardware or infrastructure on cloud or virtual environment. 

To build right thing requires continuous feedback from customers which is taken either on a prototype or directly on a product feature released to a specific set of users as part of A/B testing. Number of prototype iteration to get to the minimal viable product or desired feature and number of prototypes being worked upon in parallel are good metrics to measure continuous innovation in a team. And last but the most impactful is the number of successful new products launched.