Dimensions of Data Driven Engineering

A Data Driven Engineering organization bases its decisions and outcomes on data and information coming from their deployed systems and seeks to continually improve their solutions and delivery cadence in light of the information gathered. The Data Driven Engineering “muscle” is strong when the natural tendency for the team is to proactively seek data and information sourced from deployed systems to make decisions, and when the data and information does not exist, data sources are added with a sense of urgency to ensure good decisions are made.

I see 4 dimensions of “Intelligence” that can help engineering teams make good decisions. Consider these dimensions, debate them as needed, and build an active “Feedback Loop” and the data sources that you can utilize for your Data Driven Engineering decisions.

Solutions delivered through the cloud offer us unprecedented access to near real-time information that can be used to make timely business and engineering decisions that are increasingly being automated. This includes information on user sign up, feature adoption, customer churn, customer interest in one feature over another (A/B Testing), early warning of live site issues, discovering security breaches and automatically raising alerts and defenses, to name a few.

To get a holistic view of customer interactions and experience with the deployed cloud services, continually improve the customer experience and engineering efficiencies, and most importantly, deliver increasing value to customers, consider the following 4 dimensions of Telemetry.

Business Intelligence and Customer Intelligence telemetry focuses outwards to customers and partners. It is an opportunity to learn what is happening with the customers and how the team should respond to deliver continually increasing value with changes, new features and capabilities. When done well, the information here can have significant positive impact on revenue and margins.

Engineering Intelligence and Service Intelligence focuses inwards on what is happening with the engineering systems with a view towards increasing transaction rates, decreasing response times, and reducing failure rates. The information here can help with internal planning, cost assessment and optimization, and budgeting as well.

Business Intelligence refers to usage information received from deployed systems that provide information to make better business decisions. Typical measures include Customer Signups, Customer Inactivity (as early indicator of churn), Active User Count, Usage seasonality (to project costs), etc. The audience for this information is Finance, Business Analysts, and Product Management.

Customer Intelligence measures reflect how the solution is being used by the customers and partners, scoped to individual customers and customer segments organized across various dimensions such as size, geo, class of customer, solution combination, etc.

Customer Intelligence measures can be used to determine what features are being utilized by customers, what are not being utilized and need to be improved, which features are used together, which features should be referred to in marketing campaigns, etc. In several cases, the Customer Intelligence measures can also be shared with customers themselves to inform them of usage patterns for usage planning, demonstrate value being delivered to their user base, etc.

Customer Intelligence has some overlap with Business Intelligence and is easily confused with Business Intelligence. The key difference between the two is that customer intelligence measures reflect solution usage and are often useful for the customers’ IT departments as well. The information coming through Customer Intelligence can often be productized and delivered as additional value to partners. It is important to think of Customer Intelligence and Business Intelligence separately to ensure comprehensive telemetry coverage from the persona’s perspective.

Engineering Intelligence is the information that is consumed by the engineering team that helps them determine ongoing service health related issues, distinct service failures and failovers, resource utilization for optimization activities, internal and outside-in performance measurements, etc. These are the measures that the engineering team uses to proactively improve service delivery and performance as well as reactively utilize to address service and live site issues.

Service Intelligence consists of the measures that DevOps or Live Site Service Engineering teams use to determine service health, fire alerts on service outages and kick off automated responses, detect developing service issues, or respond to customer support incidents queries. Engineering Intelligence and Service Intelligence also tends to overlap in measures and at times used interchangeably. For true devops team, there isn’t much of a difference between these two. However, where service operations are separate from product engineering, it is better to consider Service intelligence measures independently of Engineering Intelligence.

The teams building cloud delivered solutions, specifically, the teams migrating from on-premises solutions, must consider telemetry and feedback loop as a foundational feature of the service offering. Without a highly functional near real-time telemetry system, the solutions will not achieve their full potential in the best case and have serious customer satisfaction issues in the worst case.