4 Common Pitfalls to Avoid When Tracking Engineering KPIs

This is part II of Allstacks’ KPI Best Practices Series aimed at helping engineering measure what matters, stay aligned with business objectives, and continuously improve their engineering operations. By Jeremy Freeman

In the previous article, I walked through the process of establishing outcomes-based engineering KPIs through:

• Identifying the role the engineering function plays in achieving overall company goals
• Understanding the kinds of outputs that will move the needle in achieving those goals
• Framing out the steps to get there

Sounds simple, but things can get complicated quickly when you’re juggling multiple projects, remote teams, and constantly evolving processes. This article in the KPI Best Practices Series will outline a few of the most common pitfalls in establishing an outcomes-based KPI framework, and how to avoid them.

Pitfall #1: Tracking activity instead of outcomes

Engineering KPIs need to focus on measuring outcomes, not activities, no matter how important those activities may be. This is a very easy trap to fall into. You can capture data on what your team is doing and create a compelling story of why this data shows good progress, but without tying those activities to outcomes, you can spend time and energy in the wrong place. For instance, You may track commit activity or details about a code review process, when you’re not even sure there are issues present. Delivering more code seems positive, but moving faster doesn’t always mean you are moving in the right direction. 

Instead, compile outcomes-based data in a centralized KPI dashboard, and map them back to the goals your leadership wants to accomplish. Track outcomes like shipping releases and bug rates that directly support company-wide objectives. 

Pitfall #2: Misalignment in priority

Now that outcomes—and not just activities—are tracked in your KPI framework, another mistake to avoid is around your priorities. Teams work hard to produce the best deliverables possible, but this is one of those instances where perfection can be the enemy. If you focus your energy on a single KPI or outcome, you can meet that goal, but at the expense of other outcomes. For example, your team fixes rare bugs that customers don’t often experience in an attempt to get your bug rate to zero; however, your delivery dates from your product roadmap begin to slip as a result. In this instance, you are doing great work, but you’re not working on what is most important right now.

There are two flavors of this issue. First, you discover a problem, and pivot way too hard in that direction. Second, you’re off balance in where you’re focusing, but making progress in all things. While you’re trying to meet all of your KPIs all of the time, it’s sometimes unrealistic, and you have to prioritize. It’s possible that a critical, priority one (P1) issue emerges and causes you to miss a deliverable, and at that point, you’re making the call to prioritize your application quality over your roadmap delivery. In this example, your organization prioritizes functional software over releasing new features.

All KPIs are not created equal. Having a strong sense for the priority of your KPIs is important to keep your team focused on what matters, help you make decisions about tradeoffs, and get the proper buy-in to take action accordingly. Over the medium term, revisit how much focus you place on each KPI to ensure you have a healthy culture and know when it's time to increase resources (i.e. add staff). 

KPIs should prioritize your company's overall goals instead of chasing either a mythical perfect score or just bouncing from fire to fire. Focusing on the right outcomes can help you maintain balance, expose the real needs in your organization, and protect your employees from unnecessary stress.

Pitfall #3: Existing processes hinder KPI attainment

Of course, engineering does not operate in a vacuum. Your team’s efforts towards meeting KPIs fall into some form of process. If one team doesn’t understand how their work aligns with that of other teams in the organization, there is a high risk of duplicate work, or worse, work that doesn’t get done.

Here’s just one potential scenario: A defect escalation process is put in place, where customer issues get reported to engineering as bugs. Part of that process is to assign severity and priority. Customer support assumes engineering will handle it, and engineering expects customer support to handle it. As a result, a large number of bugs get stuck in some “waiting for more information” stage of the process, and customers are not happy. The company is likely to miss its customer satisfaction goals, yet your team still claims victory because your bug SLA/ bug rate is very low, as the bugs were not taken into account. Part of defining your KPIs and tracking them back to the value they deliver will help you spot deficiencies in the KPI flow.

By planning ahead and keeping engineering KPIs aligned with goals across teams, you’re able to set expectations, establish ownership, and track the correct KPIs that keep processes working smoothly.

Pitfall #4: Important goals get lost

A corporate mission statement is a good starting point for identifying the outcomes your organization wants to achieve. However, mission statements are notoriously broad and difficult to track against. How your company responds to pressures, shifts in the market, and unforeseen crises should align with the values outlined in your mission statement. Typically, the goals embedded in those values are not well-tracked. 

Tracking KPIs that exhibit your core values can help set the tone of the organization. If one of your core values is, “We treat customers with respect, and vow to make their lives easier through the use of our product,” you should identify KPIs to track how you are making their lives easier. Engineering can develop deeper testing plans to improve product quality and alleviate frustrations, but without the accountability and measurability that good KPIs provide, tying testing plans to customer satisfaction outcomes is nearly impossible. Even if you hit your goals, you wouldn’t have the data to support that you're meeting the mission of the company. 

I suggest you have a simple, agreed upon statement that acts as a guiding light for your organization. It doesn’t have to be perfect; you just have to start somewhere. When tracking KPIs back to business outcomes, this statement can help fill in the gaps, and find where the important goals got lost.

Final Thoughts

Building well-thought-out KPIs is hard and unique to your business. It’s not once and done, but something that evolves over time. You will need to revisit your KPIs often, especially when shifts in your engineering structure stop aligning with the existing KPIs. I hope these suggestions can help you avoid some of the common pitfalls engineering teams experience when starting their KPI journey.

This blog post was originally published on Allstacks' blog