Managing Machine Learning Products - The Big Difference

If you’re a Product Manager or business leader working with machine learning (ML) for the first time, you may have searched online for guidance on how to think about and work with this capability. Most of that guidance explains the technical aspects of ML; it classifies models by type and describes the processes to create them. 

In this way, current literature treats ML as just another capability, another tool to use when addressing business problems. But there are fundamental differences when developing products which use ML. Without understanding and adapting to these differences, desirable product outcomes will remain elusive.

In this post, I will argue that the use of ML introduces a significant degree of execution risk which is not present in the development of products using traditional engineering methods. Adapting to this risk requires a change in how ML products are designed, how they are built and launched, and how product roadmaps are constructed. Failing to adapt to this risk is a major source of some of the common failures in ML product development.

Sources of Execution Risk in ML

To understand the difference in risk, it’s necessary to understand how ML products are different and why they fail.

Traditional engineering products are deterministic. When solving a business problem with these techniques, it is relatively straightforward to reason from a functional description of that problem/solution to a system design, and a team can be confident of success in creating such a system. This confidence in the creation of functional systems is captured in the Scrum Guide; the definition of a Sprint is a period of time, at the end of which a “usable product increment” is delivered.

ML products differ markedly from this; they are probabilistic. Even where the final product abstracts the probabilistic nature of the ML system, that underlying nature remains. In deciding to use a system incorporating ML you are making an implicit assumption that a probabilistic relationship can be drawn between the data you have available and the thing that you wish to predict. The complexity of that relationship is generally hidden (if it weren’t, you likely wouldn’t be using ML).

ML projects fail for a myriad of reasons. These have been well documented by my colleague Laszlo Sragner in his Data Science Fails repo, and discussed in the post Data Science Risk Categorisation. For a more in-depth discussion of risk in Machine Learning, see the linked posts.

For the remainder of this post, I want to focus on the failure modes which impact the product management approach. In particular, I will highlight four modes of failure:

• A model cannot be created which performs well enough to satisfy the original product goals. This may be due to insufficient data, or a high degree of complexity of the relationship between the data and the thing being predicted.
• A model is created which works in a controlled setting, but which fails when introducing it to production data and operating constraints.
• A model is created which satisfies the original product goals, but this does not have the anticipated impact on business success metrics.
• A product requires multiple, complex models to function effectively, and the product budget is exhausted before these can be completed successfully.

Whilst I highlight these four, which summarise a wider variety of failure modes, others are also relevant to the product approach.

Effective Strategies to Manage Risk

Given the above, how can you structure your work to mitigate this risk? How can you ensure that your products are more successful?

Lean Startup revolutionised the approach to creating new products. Using Lean, many startups manage market risk better by launching fast and exposing work early. Launching a product targeted to a small group of the market, early adopters, allows a company to better measure demand and more easily iterate to find product/market fit.

The same approach can successfully mitigate the execution risks in products which use ML. By identifying and building for a reduced-scope problem, the business impact can be measured earlier and the complexity of the wider problem space can be more accurately judged.

The ML Product Management Toolkit

Lean is a powerful tool when working with ML, but there are other key changes that you should adopt in your product approach to maximise your chance of success. I propose three key areas which should be a focus for all ML PMs:

Design ML Products For Faster Launch

As described above, taking a Lean approach will help you to better manage your product development. This involves more than just identifying a proxy “early adopter” or reducing the problem space. Your initial products must be designed to consume imperfect models, measuring performance and business impact. They should be designed to iterate on.

Identify and Track Key Business Metrics

Whilst this is the case for all products, it is particularly important for ML products. You should be clear across the entire team about the business impact you expect from the product, and you should think carefully about the metrics you will need to track to measure this impact. These metrics need to be continually monitored during the entire lifecycle of your product.

Curate a Roadmap which Balances Risk

ML roadmaps are more than the sum of their parts. Often a more complex and ambitious data product can be supported by simpler product iterations with lower risk/reward. Building a roadmap strategically, starting with small individual automation projects and growing to target more comprehensive products will build momentum in delivery and improve your team’s success rate.

In Conclusion

Many of you will have seen the headline statistics. 87% of ML projects fail. Regardless of the accuracy of this figure, it is certain that ML projects carry significant risk. Failing to adapt to the higher execution risk inherent in ML will strangle the life out of your products. You should think carefully before embarking on your next product; what can you do to reduce this waste?

I plan to follow up on the three focus areas I’ve highlighted above in the next few posts. If you’re interested in reading more, please follow me on LinkedIn, or on Twitter @ChrisHypergolic.

You can also find a more detailed and prescriptive product approach described in our book, currently available for free at machinelearningproductmanual.com.

Find out more about us at www.hypergolic.co.uk

This post has been published on www.productschool.com communities